2024-04-23T16:26:37.442 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T16:26:37.442 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776 --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} --verbose -- /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/orig.config.yaml 2024-04-23T16:26:37.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T16:26:37.568 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776 branch: wip-yuri3-testing-2024-04-05-0825 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} email: yweinste@redhat.com first_in_suite: false job_id: '7669776' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri3-testing-2024-04-05-0825 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2024-04-05-0825 sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 3753 sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2024-04-05-0825 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFDP0YXGDWK/8K5PzJgh32pqa0us48n1hYj5PEm81TzEE8Az2ZVDtbqXNkUvfKB1vdy/N7OM4zY+PQYtF04+StQ= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCerRUXaZw0MajyMm8FnD4TNG6jVXDe6qeBA40sLC16SJmuUrgYehUwaKZwbefdN1o8CcKZb6wtjQFzQtsKkufA= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_14:14:08 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T16:26:37.568 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa; will attempt to use it 2024-04-23T16:26:37.569 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks 2024-04-23T16:26:37.569 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T16:26:37.571 INFO:teuthology.task.internal:Checking packages... 2024-04-23T16:26:37.597 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5d349943c59c9485df060d6adb0594f3940ec0eb' 2024-04-23T16:26:37.597 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:26:37.597 INFO:teuthology.packaging:ref: None 2024-04-23T16:26:37.597 INFO:teuthology.packaging:tag: None 2024-04-23T16:26:37.597 INFO:teuthology.packaging:branch: wip-yuri3-testing-2024-04-05-0825 2024-04-23T16:26:37.597 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:26:37.598 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2024-04-05-0825 2024-04-23T16:26:37.738 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3211-g5d349943-1jammy 2024-04-23T16:26:37.740 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T16:26:37.746 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T16:26:37.746 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T16:26:37.790 INFO:teuthology.task.internal:Saving configuration 2024-04-23T16:26:37.804 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T16:26:37.816 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T16:26:37.845 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 16:18:49.973821', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAeIN05WIwKv7erqRFPrTttS7V8JKL6B4cmXIO+zEii3'} 2024-04-23T16:26:37.871 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 16:18:49.975192', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMFuDk0AnD1JFH+sls+1RsptPRn1TQevTx0/dquvBoKh'} 2024-04-23T16:26:37.871 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T16:26:37.877 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-23T16:26:37.877 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-23T16:26:37.877 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T16:26:37.969 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff30a90f3a0>, signals=[15]) 2024-04-23T16:26:37.970 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T16:26:37.976 INFO:teuthology.task.internal:Opening connections... 2024-04-23T16:26:37.976 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:26:37.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:26:38.057 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:26:38.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:26:38.143 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T16:26:38.149 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-04-23T16:26:38.155 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-04-23T16:26:38.156 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="Ubuntu" 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="22.04" 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_CODENAME=jammy 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:ID=ubuntu 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE=debian 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-23T16:26:38.204 INFO:teuthology.orchestra.run.smithi079.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-23T16:26:38.205 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-23T16:26:38.205 INFO:teuthology.orchestra.run.smithi079.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-23T16:26:38.205 INFO:teuthology.orchestra.run.smithi079.stdout:UBUNTU_CODENAME=jammy 2024-04-23T16:26:38.205 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-04-23T16:26:38.225 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-04-23T16:26:38.230 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-04-23T16:26:38.230 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-04-23T16:26:38.279 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="22.04" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=jammy 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-23T16:26:38.280 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=jammy 2024-04-23T16:26:38.281 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-04-23T16:26:38.300 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T16:26:38.311 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T16:26:38.335 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T16:26:38.335 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-04-23T16:26:38.338 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-04-23T16:26:38.342 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T16:26:38.366 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T16:26:38.367 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-04-23T16:26:38.385 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-04-23T16:26:38.392 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T16:26:38.434 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T16:26:38.449 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-23T16:26:38.449 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-23T16:26:38.449 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T16:26:38.450 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T16:26:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:26:38.454 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-04-23T16:26:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:26:38.457 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-04-23T16:26:38.501 INFO:teuthology.orchestra.run.smithi079.stdout:5.15.0-56-generic 2024-04-23T16:26:38.502 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.15.0-56-generic 2024-04-23T16:26:38.502 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get clean 2024-04-23T16:26:38.504 INFO:teuthology.orchestra.run.smithi204.stdout:5.15.0-56-generic 2024-04-23T16:26:38.505 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.15.0-56-generic 2024-04-23T16:26:38.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2024-04-23T16:26:38.630 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-04-23T16:26:38.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-04-23T16:26:38.771 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-23T16:26:38.883 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T16:26:38.948 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T16:26:38.948 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-23T16:26:38.965 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-23T16:26:38.999 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-23T16:26:39.003 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-23T16:26:39.094 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,388 kB] 2024-04-23T16:26:39.142 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-23T16:26:39.148 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-23T16:26:39.148 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,762 kB] 2024-04-23T16:26:39.190 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-23T16:26:39.191 INFO:teuthology.orchestra.run.smithi079.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-23T16:26:39.195 INFO:teuthology.orchestra.run.smithi079.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-23T16:26:39.196 INFO:teuthology.orchestra.run.smithi079.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-23T16:26:39.207 INFO:teuthology.orchestra.run.smithi079.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-23T16:26:39.226 INFO:teuthology.orchestra.run.smithi079.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-23T16:26:39.231 INFO:teuthology.orchestra.run.smithi079.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-23T16:26:39.234 INFO:teuthology.orchestra.run.smithi079.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-23T16:26:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-23T16:26:39.272 INFO:teuthology.orchestra.run.smithi079.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-23T16:26:39.272 INFO:teuthology.orchestra.run.smithi079.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-23T16:26:39.319 INFO:teuthology.orchestra.run.smithi079.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-23T16:26:39.360 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-23T16:26:39.386 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,388 kB] 2024-04-23T16:26:39.476 INFO:teuthology.orchestra.run.smithi079.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-23T16:26:39.499 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-23T16:26:39.711 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,608 kB] 2024-04-23T16:26:39.849 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-23T16:26:39.864 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-23T16:26:39.865 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,826 kB] 2024-04-23T16:26:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,608 kB] 2024-04-23T16:26:39.890 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-23T16:26:39.926 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-23T16:26:39.927 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [310 kB] 2024-04-23T16:26:39.935 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-23T16:26:39.958 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-23T16:26:39.968 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-23T16:26:39.981 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-23T16:26:39.982 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,762 kB] 2024-04-23T16:26:40.060 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-23T16:26:40.071 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-23T16:26:40.071 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-23T16:26:40.100 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-23T16:26:40.105 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-23T16:26:40.355 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-23T16:26:40.728 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-23T16:26:40.766 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-23T16:26:40.769 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-23T16:26:40.779 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-23T16:26:40.780 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-23T16:26:40.780 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-23T16:26:40.783 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-23T16:26:40.784 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-23T16:26:40.784 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-23T16:26:40.787 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-23T16:26:40.810 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-23T16:26:40.814 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-23T16:26:40.814 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-23T16:26:40.814 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-23T16:26:40.815 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-23T16:26:40.981 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-23T16:26:41.048 INFO:teuthology.orchestra.run.smithi079.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-23T16:26:41.064 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-23T16:26:41.165 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-23T16:26:41.188 INFO:teuthology.orchestra.run.smithi079.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-23T16:26:41.192 INFO:teuthology.orchestra.run.smithi079.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,826 kB] 2024-04-23T16:26:41.228 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-23T16:26:41.255 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-23T16:26:41.753 INFO:teuthology.orchestra.run.smithi079.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-23T16:26:41.759 INFO:teuthology.orchestra.run.smithi079.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [310 kB] 2024-04-23T16:26:41.812 INFO:teuthology.orchestra.run.smithi079.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-23T16:26:42.005 INFO:teuthology.orchestra.run.smithi079.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-23T16:26:42.936 INFO:teuthology.orchestra.run.smithi079.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-23T16:26:43.025 INFO:teuthology.orchestra.run.smithi079.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-23T16:26:43.046 INFO:teuthology.orchestra.run.smithi079.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-23T16:26:43.049 INFO:teuthology.orchestra.run.smithi079.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-23T16:26:43.052 INFO:teuthology.orchestra.run.smithi079.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-23T16:26:43.053 INFO:teuthology.orchestra.run.smithi079.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-23T16:26:43.054 INFO:teuthology.orchestra.run.smithi079.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-23T16:26:43.054 INFO:teuthology.orchestra.run.smithi079.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-23T16:26:43.059 INFO:teuthology.orchestra.run.smithi079.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-23T16:26:43.069 INFO:teuthology.orchestra.run.smithi079.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-23T16:26:43.237 INFO:teuthology.orchestra.run.smithi079.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-23T16:26:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-23T16:26:43.422 INFO:teuthology.orchestra.run.smithi079.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-23T16:26:43.482 INFO:teuthology.orchestra.run.smithi079.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-23T16:26:43.513 INFO:teuthology.orchestra.run.smithi079.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-23T16:26:45.505 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 13.2 MB in 3s (5,224 kB/s) 2024-04-23T16:26:47.067 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-04-23T16:26:47.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-23T16:26:47.149 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-04-23T16:26:47.349 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-04-23T16:26:47.350 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-04-23T16:26:47.527 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T16:26:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T16:26:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T16:26:47.528 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T16:26:47.528 INFO:teuthology.orchestra.run.smithi204.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-23T16:26:47.528 INFO:teuthology.orchestra.run.smithi204.stdout: python2.7-minimal 2024-04-23T16:26:47.528 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T16:26:47.529 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2024-04-23T16:26:47.529 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-23T16:26:47.529 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-23T16:26:47.530 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-23T16:26:47.531 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2024-04-23T16:26:47.531 INFO:teuthology.orchestra.run.smithi204.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-23T16:26:47.531 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2024-04-23T16:26:47.531 INFO:teuthology.orchestra.run.smithi204.stdout: thermald 2024-04-23T16:26:47.584 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2024-04-23T16:26:47.584 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-23T16:26:47.584 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-23T16:26:47.585 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-23T16:26:47.586 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2024-04-23T16:26:47.586 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-23T16:26:47.648 INFO:teuthology.orchestra.run.smithi204.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-23T16:26:47.648 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 113 MB of archives. 2024-04-23T16:26:47.648 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-23T16:26:47.648 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-23T16:26:47.815 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 13.2 MB in 5s (2,743 kB/s) 2024-04-23T16:26:49.105 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-04-23T16:26:49.122 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-23T16:26:49.181 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-04-23T16:26:49.381 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-04-23T16:26:49.382 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-04-23T16:26:49.557 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T16:26:49.558 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T16:26:49.558 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T16:26:49.558 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T16:26:49.558 INFO:teuthology.orchestra.run.smithi079.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-23T16:26:49.558 INFO:teuthology.orchestra.run.smithi079.stdout: python2.7-minimal 2024-04-23T16:26:49.559 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T16:26:49.560 INFO:teuthology.orchestra.run.smithi079.stdout:The following additional packages will be installed: 2024-04-23T16:26:49.560 INFO:teuthology.orchestra.run.smithi079.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-23T16:26:49.560 INFO:teuthology.orchestra.run.smithi079.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-23T16:26:49.560 INFO:teuthology.orchestra.run.smithi079.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-23T16:26:49.561 INFO:teuthology.orchestra.run.smithi079.stdout:Suggested packages: 2024-04-23T16:26:49.562 INFO:teuthology.orchestra.run.smithi079.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-23T16:26:49.562 INFO:teuthology.orchestra.run.smithi079.stdout:Recommended packages: 2024-04-23T16:26:49.562 INFO:teuthology.orchestra.run.smithi079.stdout: thermald 2024-04-23T16:26:49.592 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-23T16:26:49.620 INFO:teuthology.orchestra.run.smithi079.stdout:The following NEW packages will be installed: 2024-04-23T16:26:49.620 INFO:teuthology.orchestra.run.smithi079.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-23T16:26:49.621 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-23T16:26:49.621 INFO:teuthology.orchestra.run.smithi079.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-23T16:26:49.622 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages will be upgraded: 2024-04-23T16:26:49.622 INFO:teuthology.orchestra.run.smithi079.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-23T16:26:49.802 INFO:teuthology.orchestra.run.smithi079.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-23T16:26:49.802 INFO:teuthology.orchestra.run.smithi079.stdout:Need to get 113 MB of archives. 2024-04-23T16:26:49.802 INFO:teuthology.orchestra.run.smithi079.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-23T16:26:49.802 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-23T16:26:50.245 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-23T16:26:51.210 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-23T16:26:51.559 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-23T16:26:53.830 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-23T16:26:53.830 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-23T16:26:53.831 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-23T16:26:54.319 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-23T16:26:54.374 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-23T16:26:54.828 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 113 MB in 5s (23.9 MB/s) 2024-04-23T16:26:55.020 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-23T16:26:57.059 INFO:teuthology.orchestra.run.smithi079.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-23T16:26:57.064 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:26:57.125 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:26:58.133 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-23T16:26:58.133 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-23T16:26:58.134 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-23T16:26:59.477 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-23T16:26:59.498 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:26:59.552 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-23T16:26:59.570 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:26:59.611 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-23T16:27:00.010 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 113 MB in 12s (9,427 kB/s) 2024-04-23T16:27:00.013 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-23T16:27:00.034 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:27:00.063 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:00.212 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-23T16:27:02.214 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-23T16:27:02.218 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:27:02.284 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:04.469 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-23T16:27:04.490 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:27:04.589 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:04.998 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-23T16:27:05.018 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:27:05.056 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:06.387 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T16:27:06.506 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T16:27:06.823 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T16:27:06.951 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T16:27:07.238 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-23T16:27:07.260 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-23T16:27:07.304 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-23T16:27:12.090 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T16:27:12.209 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T16:27:12.525 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T16:27:12.646 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T16:27:12.940 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-23T16:27:12.963 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-23T16:27:12.999 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-23T16:27:13.022 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-23T16:27:13.059 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:27:13.097 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:16.443 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T16:27:16.566 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T16:27:16.907 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-23T16:27:17.004 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:17.122 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-23T16:27:17.250 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:18.558 INFO:teuthology.orchestra.run.smithi079.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-23T16:27:18.558 INFO:teuthology.orchestra.run.smithi079.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-23T16:27:18.558 INFO:teuthology.orchestra.run.smithi079.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-23T16:27:18.558 INFO:teuthology.orchestra.run.smithi079.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:18.718 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:19.322 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-23T16:27:19.359 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T16:27:19.397 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:20.112 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:21.471 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-23T16:27:21.597 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-23T16:27:21.724 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:21.780 INFO:teuthology.orchestra.run.smithi079.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-23T16:27:21.780 INFO:teuthology.orchestra.run.smithi079.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:23.109 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T16:27:23.234 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T16:27:23.575 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-23T16:27:23.689 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:23.815 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-23T16:27:23.935 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:25.230 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-23T16:27:25.230 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-23T16:27:25.230 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-23T16:27:25.230 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:25.388 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:26.781 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:28.149 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-23T16:27:28.266 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-23T16:27:28.393 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T16:27:28.463 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-23T16:27:28.463 INFO:teuthology.orchestra.run.smithi204.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:36.049 INFO:teuthology.orchestra.run.smithi079.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-23T16:27:36.049 INFO:teuthology.orchestra.run.smithi079.stdout:Sourcing file `/etc/default/grub' 2024-04-23T16:27:36.073 INFO:teuthology.orchestra.run.smithi079.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-23T16:27:36.096 INFO:teuthology.orchestra.run.smithi079.stdout:Generating grub configuration file ... 2024-04-23T16:27:36.354 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-23T16:27:36.379 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:36.645 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-23T16:27:36.651 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-23T16:27:36.719 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-23T16:27:36.724 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-23T16:27:36.871 INFO:teuthology.orchestra.run.smithi079.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-23T16:27:36.871 INFO:teuthology.orchestra.run.smithi079.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-23T16:27:36.871 INFO:teuthology.orchestra.run.smithi079.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-23T16:27:36.904 INFO:teuthology.orchestra.run.smithi079.stdout:done 2024-04-23T16:27:37.378 DEBUG:teuthology.orchestra.run.smithi079:> dpkg -s linux-image-generic 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Package: linux-image-generic 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Status: install ok installed 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Priority: optional 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Section: kernel 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Installed-Size: 21 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Maintainer: Ubuntu Kernel Team 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: amd64 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Source: linux-meta 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Version: 5.15.0.105.102 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-23T16:27:37.399 INFO:teuthology.orchestra.run.smithi079.stdout:Recommends: thermald 2024-04-23T16:27:37.400 INFO:teuthology.orchestra.run.smithi079.stdout:Description: Generic Linux kernel image 2024-04-23T16:27:37.400 INFO:teuthology.orchestra.run.smithi079.stdout: This package will always depend on the latest generic kernel image 2024-04-23T16:27:37.400 INFO:teuthology.orchestra.run.smithi079.stdout: available. 2024-04-23T16:27:37.400 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-23T16:27:37.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-23T16:27:37.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-23T16:27:37.400 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-23T16:27:37.400 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-04-23T16:27:37.401 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get clean 2024-04-23T16:27:37.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-04-23T16:27:37.741 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-23T16:27:37.741 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T16:27:37.754 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-23T16:27:37.778 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-23T16:27:39.102 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-04-23T16:27:39.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-23T16:27:39.182 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-04-23T16:27:39.383 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-04-23T16:27:39.383 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-04-23T16:27:39.560 INFO:teuthology.orchestra.run.smithi079.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-23T16:27:39.560 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T16:27:39.560 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T16:27:39.560 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T16:27:39.560 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T16:27:39.561 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-23T16:27:39.561 INFO:teuthology.orchestra.run.smithi079.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-23T16:27:39.561 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T16:27:39.604 INFO:teuthology.orchestra.run.smithi079.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-23T16:27:39.606 DEBUG:teuthology.orchestra.run.smithi079:> dpkg -s linux-image-generic 2024-04-23T16:27:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:Package: linux-image-generic 2024-04-23T16:27:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:Status: install ok installed 2024-04-23T16:27:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:Priority: optional 2024-04-23T16:27:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:Section: kernel 2024-04-23T16:27:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:Installed-Size: 21 2024-04-23T16:27:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:Maintainer: Ubuntu Kernel Team 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: amd64 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Source: linux-meta 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Version: 5.15.0.105.102 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Recommends: thermald 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout:Description: Generic Linux kernel image 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout: This package will always depend on the latest generic kernel image 2024-04-23T16:27:39.622 INFO:teuthology.orchestra.run.smithi079.stdout: available. 2024-04-23T16:27:39.623 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-04-23T16:27:39.671 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.DPhsCVM9Pq 2024-04-23T16:27:39.671 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub/grub.cfg /tmp/tmp.DPhsCVM9Pq 2024-04-23T16:27:39.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.DPhsCVM9Pq 2024-04-23T16:27:39.856 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.DPhsCVM9Pq is 10KB 2024-04-23T16:27:39.910 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.DPhsCVM9Pq 2024-04-23T16:27:39.916 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-23T16:27:39.972 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:27:39.972 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-23T16:27:39.973 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-23T16:27:40.047 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-23T16:27:40.047 DEBUG:teuthology.orchestra.run.smithi079:> sudo update-grub 2024-04-23T16:27:41.290 INFO:teuthology.orchestra.run.smithi079.stderr:Sourcing file `/etc/default/grub' 2024-04-23T16:27:41.291 INFO:teuthology.orchestra.run.smithi079.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-23T16:27:41.292 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-04-23T16:27:41.468 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-23T16:27:41.474 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:41.689 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-23T16:27:41.694 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-23T16:27:41.733 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-23T16:27:41.737 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-23T16:27:41.859 INFO:teuthology.orchestra.run.smithi079.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-23T16:27:41.859 INFO:teuthology.orchestra.run.smithi079.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-23T16:27:41.859 INFO:teuthology.orchestra.run.smithi079.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-23T16:27:41.872 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-04-23T16:27:41.881 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-04-23T16:27:42.437 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-23T16:27:42.437 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub' 2024-04-23T16:27:42.449 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-23T16:27:42.475 INFO:teuthology.orchestra.run.smithi204.stdout:Generating grub configuration file ... 2024-04-23T16:27:42.883 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-23T16:27:42.905 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:43.241 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-23T16:27:43.246 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-23T16:27:43.314 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-23T16:27:43.320 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-23T16:27:43.473 INFO:teuthology.orchestra.run.smithi204.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-23T16:27:43.473 INFO:teuthology.orchestra.run.smithi204.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-23T16:27:43.473 INFO:teuthology.orchestra.run.smithi204.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-23T16:27:43.510 INFO:teuthology.orchestra.run.smithi204.stdout:done 2024-04-23T16:27:43.935 DEBUG:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 21 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2024-04-23T16:27:43.957 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 5.15.0.105.102 2024-04-23T16:27:43.958 INFO:teuthology.orchestra.run.smithi204.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-23T16:27:43.958 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-23T16:27:43.958 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2024-04-23T16:27:43.958 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2024-04-23T16:27:43.958 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2024-04-23T16:27:43.958 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2024-04-23T16:27:43.958 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-23T16:27:43.958 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-23T16:27:43.958 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-23T16:27:43.958 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-23T16:27:43.959 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-04-23T16:27:43.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2024-04-23T16:27:44.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-04-23T16:27:44.238 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T16:27:44.249 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-23T16:27:44.269 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-23T16:27:44.351 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-23T16:27:45.604 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-04-23T16:27:45.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-23T16:27:45.684 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-04-23T16:27:45.891 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-04-23T16:27:45.892 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-04-23T16:27:46.071 INFO:teuthology.orchestra.run.smithi204.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-23T16:27:46.071 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T16:27:46.072 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T16:27:46.072 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T16:27:46.072 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T16:27:46.072 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-23T16:27:46.072 INFO:teuthology.orchestra.run.smithi204.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-23T16:27:46.073 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T16:27:46.114 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-23T16:27:46.116 DEBUG:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 21 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 5.15.0.105.102 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-23T16:27:46.128 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-23T16:27:46.129 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2024-04-23T16:27:46.129 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2024-04-23T16:27:46.129 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2024-04-23T16:27:46.129 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2024-04-23T16:27:46.129 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-04-23T16:27:46.180 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.Q3C4G2mpBu 2024-04-23T16:27:46.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Q3C4G2mpBu 2024-04-23T16:27:46.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.Q3C4G2mpBu 2024-04-23T16:27:46.355 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.Q3C4G2mpBu is 10KB 2024-04-23T16:27:46.367 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.Q3C4G2mpBu 2024-04-23T16:27:46.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-23T16:27:46.432 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:27:46.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-23T16:27:46.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-23T16:27:46.499 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-23T16:27:46.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo update-grub 2024-04-23T16:27:47.701 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub' 2024-04-23T16:27:47.702 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-23T16:27:47.704 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-04-23T16:27:47.883 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-23T16:27:47.889 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-23T16:27:48.120 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-23T16:27:48.124 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-23T16:27:48.162 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-23T16:27:48.166 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-23T16:27:48.280 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-23T16:27:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-23T16:27:48.281 INFO:teuthology.orchestra.run.smithi204.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-23T16:27:48.301 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-04-23T16:27:48.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-04-23T16:28:11.906 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-23T16:28:11.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:28:11.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:28:18.311 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-23T16:28:18.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:28:18.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:28:30.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-04-23T16:28:36.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-04-23T16:28:39.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:28:39.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:28:42.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-04-23T16:28:45.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:28:45.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:28:48.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-04-23T16:28:54.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:28:54.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:29:00.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:29:00.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:29:12.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-04-23T16:29:19.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-04-23T16:29:27.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:29:27.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:29:28.165 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-04-23T16:29:28.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:29:28.762 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-23T16:29:28.762 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-04-23T16:29:28.809 INFO:teuthology.orchestra.run.smithi079.stdout:5.15.0-105-generic 2024-04-23T16:29:28.809 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-23T16:29:28.809 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T16:29:28.809 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-23T16:29:29.810 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-23T16:29:29.810 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T16:29:29.880 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-04-23T16:29:29.902 DEBUG:teuthology.parallel:result is None 2024-04-23T16:29:34.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:29:34.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:29:34.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-04-23T16:29:52.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:29:52.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:29:52.541 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-04-23T16:29:53.055 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:29:53.055 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-23T16:29:53.055 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-04-23T16:29:53.101 INFO:teuthology.orchestra.run.smithi204.stdout:5.15.0-105-generic 2024-04-23T16:29:53.101 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-23T16:29:53.101 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T16:29:53.101 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-23T16:29:54.102 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-23T16:29:54.102 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T16:29:54.185 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-04-23T16:29:54.219 DEBUG:teuthology.parallel:result is None 2024-04-23T16:29:54.219 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T16:29:54.226 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T16:29:54.226 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T16:29:54.229 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T16:29:54.234 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T16:29:54.264 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T16:29:54.297 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T16:29:54.297 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T16:29:54.300 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T16:29:54.320 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T16:29:54.331 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T16:29:54.331 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T16:29:54.359 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T16:29:54.379 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T16:29:54.387 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T16:29:54.396 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T16:29:54.405 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T16:29:54.407 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T16:29:54.414 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T16:29:54.414 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T16:29:54.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T16:29:54.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T16:29:54.479 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T16:29:54.479 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T16:29:54.513 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T16:29:54.519 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T16:29:54.565 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T16:29:54.613 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:29:54.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T16:29:54.686 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T16:29:54.692 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T16:29:54.737 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:29:54.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T16:29:54.793 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-04-23T16:29:54.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-04-23T16:29:54.865 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T16:29:54.873 INFO:teuthology.task.internal:Starting timer... 2024-04-23T16:29:54.874 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T16:29:54.910 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T16:29:54.948 DEBUG:teuthology.task.selinux:Excluding smithi079: OS 'ubuntu' does not support SELinux 2024-04-23T16:29:54.949 DEBUG:teuthology.task.selinux:Excluding smithi204: OS 'ubuntu' does not support SELinux 2024-04-23T16:29:54.949 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T16:29:54.949 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-23T16:29:54.949 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T16:29:54.950 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T16:29:54.976 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T16:29:54.995 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T16:29:54.996 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T16:36:59.510 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-04-23T16:36:59.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:36:59.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:36:59.605 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-04-23T16:36:59.687 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-23T16:36:59.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:36:59.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:36:59.771 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-04-23T16:36:59.850 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T16:36:59.850 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T16:36:59.862 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T16:36:59.862 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T16:36:59.863 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:36:59.865 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T16:36:59.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:36:59.896 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Command line: ntpd -gq 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: ---------------------------------------------------- 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: ntp-4 is maintained by Network Time Foundation, 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: corporation. Support and training for ntp-4 are 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: available at https://www.nwtime.org/support 2024-04-23T16:36:59.897 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: ---------------------------------------------------- 2024-04-23T16:36:59.898 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: proto: precision = 0.059 usec (-24) 2024-04-23T16:36:59.898 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: basedate set to 2022-02-04 2024-04-23T16:36:59.898 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: gps base set to 2022-02-06 (week 2196) 2024-04-23T16:36:59.898 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-23T16:36:59.898 INFO:teuthology.orchestra.run.smithi079.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-23T16:36:59.898 INFO:teuthology.orchestra.run.smithi079.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-23T16:36:59.899 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: restrict ::: KOD does nothing without LIMITED. 2024-04-23T16:36:59.899 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listen and drop on 0 v6wildcard [::]:123 2024-04-23T16:36:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-23T16:36:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listen normally on 2 lo 127.0.0.1:123 2024-04-23T16:36:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listen normally on 3 enp3s0f1 172.21.15.79:123 2024-04-23T16:36:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listen normally on 4 lo [::1]:123 2024-04-23T16:36:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce87%5]:123 2024-04-23T16:36:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:36:59 ntpd[17237]: Listening on routing socket on fd #22 for interface updates 2024-04-23T16:36:59.916 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Command line: ntpd -gq 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: ---------------------------------------------------- 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: ntp-4 is maintained by Network Time Foundation, 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: corporation. Support and training for ntp-4 are 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: available at https://www.nwtime.org/support 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: ---------------------------------------------------- 2024-04-23T16:36:59.917 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: proto: precision = 0.044 usec (-24) 2024-04-23T16:36:59.918 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: basedate set to 2022-02-04 2024-04-23T16:36:59.918 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: gps base set to 2022-02-06 (week 2196) 2024-04-23T16:36:59.918 INFO:teuthology.orchestra.run.smithi204.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-23T16:36:59.918 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-23T16:36:59.918 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: restrict ::: KOD does nothing without LIMITED. 2024-04-23T16:36:59.919 INFO:teuthology.orchestra.run.smithi204.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-23T16:36:59.919 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listen and drop on 0 v6wildcard [::]:123 2024-04-23T16:36:59.920 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-23T16:36:59.920 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listen normally on 2 lo 127.0.0.1:123 2024-04-23T16:36:59.920 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listen normally on 3 enp3s0f1 172.21.15.204:123 2024-04-23T16:36:59.920 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listen normally on 4 lo [::1]:123 2024-04-23T16:36:59.920 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ab%5]:123 2024-04-23T16:36:59.920 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:36:59 ntpd[17197]: Listening on routing socket on fd #22 for interface updates 2024-04-23T16:37:00.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:00 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:00.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:00 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:00.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:00 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:00.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:00 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:00.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:00 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:00.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:00 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:00.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:00 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:00.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:00 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:00.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:00 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:00.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:00 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:00.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:00 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:00.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:00 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:01.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:01 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:01.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:01 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:01.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:01 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:01.917 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:01 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:02 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:02 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:02 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:02 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:02 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:02.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:02 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:02.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:02.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:02.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:02.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:02 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:03.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:03 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:03.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:03 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:03.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:03 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:03.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:03 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:03.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:03 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:03.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:03 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:04.899 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:04 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:04.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:04.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:04.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:04 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:05.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:05 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:05.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:05 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:05.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:05 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:05.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:05 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:05.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:05 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:05.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:05 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:06.897 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:06.898 INFO:teuthology.orchestra.run.smithi079.stdout:23 Apr 16:37:06 ntpd[17237]: ntpd: time slew -0.001697 s 2024-04-23T16:37:06.899 INFO:teuthology.orchestra.run.smithi079.stdout:ntpd: time slew -0.001697s 2024-04-23T16:37:06.899 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-23T16:37:06.899 INFO:teuthology.orchestra.run.smithi079.stderr:23 Apr 16:37:06 ntpd[17237]: can't open /var/log/ntpstats/loopstats.20240423: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:06.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:06 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:06.950 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2024-04-23T16:37:06.950 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2024-04-23T16:37:06.950 INFO:teuthology.orchestra.run.smithi079.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:06.950 INFO:teuthology.orchestra.run.smithi079.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:06.950 INFO:teuthology.orchestra.run.smithi079.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:06.950 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:08.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:08.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:08.918 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stdout:23 Apr 16:37:08 ntpd[17197]: ntpd: time slew -0.000866 s 2024-04-23T16:37:08.919 INFO:teuthology.orchestra.run.smithi204.stdout:ntpd: time slew -0.000866s 2024-04-23T16:37:08.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-23T16:37:08.920 INFO:teuthology.orchestra.run.smithi204.stderr:23 Apr 16:37:08 ntpd[17197]: can't open /var/log/ntpstats/loopstats.20240423: Permission denied 2024-04-23T16:37:08.967 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2024-04-23T16:37:08.967 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2024-04-23T16:37:08.967 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:08.967 INFO:teuthology.orchestra.run.smithi204.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:08.967 INFO:teuthology.orchestra.run.smithi204.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:08.967 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:37:08.968 INFO:teuthology.run_tasks:Running task install... 2024-04-23T16:37:08.977 DEBUG:teuthology.task.install:project ceph 2024-04-23T16:37:08.977 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}} 2024-04-23T16:37:08.978 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'} 2024-04-23T16:37:08.978 INFO:teuthology.task.install:Using flavor: default 2024-04-23T16:37:08.983 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T16:37:08.984 INFO:teuthology.task.install:extra packages: [] 2024-04-23T16:37:08.984 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-key list | grep Ceph 2024-04-23T16:37:08.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-key list | grep Ceph 2024-04-23T16:37:09.044 INFO:teuthology.orchestra.run.smithi079.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-23T16:37:09.064 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-23T16:37:09.090 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-23T16:37:09.091 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph.com (release key) 2024-04-23T16:37:09.091 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-23T16:37:09.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:37:09.097 INFO:teuthology.orchestra.run.smithi079.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-23T16:37:09.097 INFO:teuthology.orchestra.run.smithi079.stdout:uid [ unknown] Ceph.com (release key) 2024-04-23T16:37:09.098 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-23T16:37:09.098 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:37:09.246 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default/ 2024-04-23T16:37:09.247 INFO:teuthology.task.install.deb:Package version is 19.0.0-3211-g5d349943-1jammy 2024-04-23T16:37:09.321 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default/ 2024-04-23T16:37:09.505 INFO:teuthology.task.install.deb:Package version is 19.0.0-3211-g5d349943-1jammy 2024-04-23T16:37:09.622 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:37:09.622 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-23T16:37:09.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-04-23T16:37:09.680 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:37:09.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-23T16:37:09.695 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-04-23T16:37:09.876 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-23T16:37:09.918 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T16:37:09.918 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-23T16:37:09.930 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-23T16:37:09.931 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T16:37:09.955 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-23T16:37:09.982 INFO:teuthology.orchestra.run.smithi079.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy InRelease 2024-04-23T16:37:10.013 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-23T16:37:10.018 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy InRelease 2024-04-23T16:37:10.019 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-23T16:37:10.064 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-23T16:37:10.065 INFO:teuthology.orchestra.run.smithi079.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-23T16:37:10.095 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-23T16:37:10.111 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2024-04-23T16:37:10.111 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-23T16:37:10.154 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-23T16:37:10.155 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-23T16:37:10.195 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2024-04-23T16:37:10.267 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 29.2 kB in 0s (64.9 kB/s) 2024-04-23T16:37:10.270 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 29.2 kB in 0s (73.5 kB/s) 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-04-23T16:37:11.235 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3211-g5d349943-1jammy cephadm=19.0.0-3211-g5d349943-1jammy ceph-mds=19.0.0-3211-g5d349943-1jammy ceph-mgr=19.0.0-3211-g5d349943-1jammy ceph-common=19.0.0-3211-g5d349943-1jammy ceph-fuse=19.0.0-3211-g5d349943-1jammy ceph-test=19.0.0-3211-g5d349943-1jammy ceph-volume=19.0.0-3211-g5d349943-1jammy radosgw=19.0.0-3211-g5d349943-1jammy python3-rados=19.0.0-3211-g5d349943-1jammy python3-rgw=19.0.0-3211-g5d349943-1jammy python3-cephfs=19.0.0-3211-g5d349943-1jammy python3-rbd=19.0.0-3211-g5d349943-1jammy libcephfs2=19.0.0-3211-g5d349943-1jammy libcephfs-dev=19.0.0-3211-g5d349943-1jammy librados2=19.0.0-3211-g5d349943-1jammy librbd1=19.0.0-3211-g5d349943-1jammy rbd-fuse=19.0.0-3211-g5d349943-1jammy 2024-04-23T16:37:11.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3211-g5d349943-1jammy cephadm=19.0.0-3211-g5d349943-1jammy ceph-mds=19.0.0-3211-g5d349943-1jammy ceph-mgr=19.0.0-3211-g5d349943-1jammy ceph-common=19.0.0-3211-g5d349943-1jammy ceph-fuse=19.0.0-3211-g5d349943-1jammy ceph-test=19.0.0-3211-g5d349943-1jammy ceph-volume=19.0.0-3211-g5d349943-1jammy radosgw=19.0.0-3211-g5d349943-1jammy python3-rados=19.0.0-3211-g5d349943-1jammy python3-rgw=19.0.0-3211-g5d349943-1jammy python3-cephfs=19.0.0-3211-g5d349943-1jammy python3-rbd=19.0.0-3211-g5d349943-1jammy libcephfs2=19.0.0-3211-g5d349943-1jammy libcephfs-dev=19.0.0-3211-g5d349943-1jammy librados2=19.0.0-3211-g5d349943-1jammy librbd1=19.0.0-3211-g5d349943-1jammy rbd-fuse=19.0.0-3211-g5d349943-1jammy 2024-04-23T16:37:11.289 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-04-23T16:37:11.289 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-04-23T16:37:11.465 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-04-23T16:37:11.466 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-04-23T16:37:11.466 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-04-23T16:37:11.467 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-04-23T16:37:11.638 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T16:37:11.638 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T16:37:11.638 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T16:37:11.638 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T16:37:11.639 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-23T16:37:11.639 INFO:teuthology.orchestra.run.smithi079.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-23T16:37:11.639 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-23T16:37:11.641 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T16:37:11.642 INFO:teuthology.orchestra.run.smithi079.stdout:The following additional packages will be installed: 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-23T16:37:11.646 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet zip 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout:Suggested packages: 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-23T16:37:11.647 INFO:teuthology.orchestra.run.smithi079.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout:Recommended packages: 2024-04-23T16:37:11.648 INFO:teuthology.orchestra.run.smithi079.stdout: btrfs-tools 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-23T16:37:11.650 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet zip 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2024-04-23T16:37:11.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2024-04-23T16:37:11.652 INFO:teuthology.orchestra.run.smithi204.stdout: btrfs-tools 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout:The following NEW packages will be installed: 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-23T16:37:11.741 INFO:teuthology.orchestra.run.smithi079.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-23T16:37:11.744 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-23T16:37:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-23T16:37:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-23T16:37:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-23T16:37:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-23T16:37:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-23T16:37:11.745 INFO:teuthology.orchestra.run.smithi079.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-23T16:37:11.746 INFO:teuthology.orchestra.run.smithi204.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-23T16:37:11.747 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-23T16:37:11.748 INFO:teuthology.orchestra.run.smithi204.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-23T16:37:11.807 INFO:teuthology.orchestra.run.smithi079.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-23T16:37:11.807 INFO:teuthology.orchestra.run.smithi079.stdout:Need to get 181 MB of archives. 2024-04-23T16:37:11.807 INFO:teuthology.orchestra.run.smithi079.stdout:After this operation, 828 MB of additional disk space will be used. 2024-04-23T16:37:11.807 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-23T16:37:11.808 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-23T16:37:11.808 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 181 MB of archives. 2024-04-23T16:37:11.808 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 828 MB of additional disk space will be used. 2024-04-23T16:37:11.809 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-23T16:37:11.883 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3211-g5d349943-1jammy [3,554 kB] 2024-04-23T16:37:11.885 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3211-g5d349943-1jammy [3,554 kB] 2024-04-23T16:37:11.910 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-23T16:37:11.915 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-23T16:37:11.925 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-23T16:37:11.927 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-23T16:37:11.933 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-23T16:37:11.938 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-23T16:37:11.998 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-23T16:37:12.002 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-23T16:37:12.006 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-23T16:37:12.011 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-23T16:37:12.015 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-23T16:37:12.019 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-23T16:37:12.020 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-23T16:37:12.021 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-23T16:37:12.025 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-23T16:37:12.030 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-23T16:37:12.030 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-23T16:37:12.030 INFO:teuthology.orchestra.run.smithi079.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-23T16:37:12.031 INFO:teuthology.orchestra.run.smithi079.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-23T16:37:12.032 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-23T16:37:12.034 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-23T16:37:12.037 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-23T16:37:12.041 INFO:teuthology.orchestra.run.smithi079.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-23T16:37:12.044 INFO:teuthology.orchestra.run.smithi079.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-23T16:37:12.047 INFO:teuthology.orchestra.run.smithi079.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-23T16:37:12.048 INFO:teuthology.orchestra.run.smithi079.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-23T16:37:12.055 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-23T16:37:12.057 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-23T16:37:12.060 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-23T16:37:12.062 INFO:teuthology.orchestra.run.smithi079.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-23T16:37:12.062 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-23T16:37:12.062 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-23T16:37:12.062 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-23T16:37:12.064 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-23T16:37:12.064 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-23T16:37:12.064 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-23T16:37:12.065 INFO:teuthology.orchestra.run.smithi079.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-23T16:37:12.067 INFO:teuthology.orchestra.run.smithi079.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-23T16:37:12.069 INFO:teuthology.orchestra.run.smithi079.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-23T16:37:12.070 INFO:teuthology.orchestra.run.smithi079.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-23T16:37:12.079 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-23T16:37:12.082 INFO:teuthology.orchestra.run.smithi079.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-23T16:37:12.084 INFO:teuthology.orchestra.run.smithi079.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-23T16:37:12.084 INFO:teuthology.orchestra.run.smithi079.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-23T16:37:12.084 INFO:teuthology.orchestra.run.smithi079.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-23T16:37:12.085 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-23T16:37:12.085 INFO:teuthology.orchestra.run.smithi079.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-23T16:37:12.085 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-23T16:37:12.085 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-23T16:37:12.088 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-23T16:37:12.089 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-23T16:37:12.089 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-23T16:37:12.090 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-23T16:37:12.091 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-23T16:37:12.098 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-23T16:37:12.103 INFO:teuthology.orchestra.run.smithi079.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-23T16:37:12.103 INFO:teuthology.orchestra.run.smithi079.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-23T16:37:12.104 INFO:teuthology.orchestra.run.smithi079.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-23T16:37:12.107 INFO:teuthology.orchestra.run.smithi079.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-23T16:37:12.108 INFO:teuthology.orchestra.run.smithi079.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-23T16:37:12.111 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-23T16:37:12.113 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-23T16:37:12.113 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-23T16:37:12.114 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-23T16:37:12.119 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-23T16:37:12.119 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-23T16:37:12.120 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-23T16:37:12.121 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-23T16:37:12.122 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-23T16:37:12.123 INFO:teuthology.orchestra.run.smithi079.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-23T16:37:12.124 INFO:teuthology.orchestra.run.smithi079.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-23T16:37:12.125 INFO:teuthology.orchestra.run.smithi079.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-23T16:37:12.125 INFO:teuthology.orchestra.run.smithi079.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-23T16:37:12.126 INFO:teuthology.orchestra.run.smithi079.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-23T16:37:12.138 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-23T16:37:12.139 INFO:teuthology.orchestra.run.smithi204.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-23T16:37:12.141 INFO:teuthology.orchestra.run.smithi204.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-23T16:37:12.142 INFO:teuthology.orchestra.run.smithi204.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-23T16:37:12.144 INFO:teuthology.orchestra.run.smithi079.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-23T16:37:12.145 INFO:teuthology.orchestra.run.smithi079.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-23T16:37:12.145 INFO:teuthology.orchestra.run.smithi079.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-23T16:37:12.152 INFO:teuthology.orchestra.run.smithi079.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-23T16:37:12.153 INFO:teuthology.orchestra.run.smithi079.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-23T16:37:12.159 INFO:teuthology.orchestra.run.smithi204.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-23T16:37:12.160 INFO:teuthology.orchestra.run.smithi204.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-23T16:37:12.161 INFO:teuthology.orchestra.run.smithi204.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-23T16:37:12.163 INFO:teuthology.orchestra.run.smithi204.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-23T16:37:12.173 INFO:teuthology.orchestra.run.smithi079.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-23T16:37:12.174 INFO:teuthology.orchestra.run.smithi079.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-23T16:37:12.174 INFO:teuthology.orchestra.run.smithi079.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3211-g5d349943-1jammy [3,223 kB] 2024-04-23T16:37:12.175 INFO:teuthology.orchestra.run.smithi079.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-23T16:37:12.178 INFO:teuthology.orchestra.run.smithi079.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-23T16:37:12.179 INFO:teuthology.orchestra.run.smithi079.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-23T16:37:12.212 INFO:teuthology.orchestra.run.smithi079.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-23T16:37:12.213 INFO:teuthology.orchestra.run.smithi079.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-23T16:37:12.218 INFO:teuthology.orchestra.run.smithi079.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-23T16:37:12.220 INFO:teuthology.orchestra.run.smithi079.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-23T16:37:12.221 INFO:teuthology.orchestra.run.smithi079.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-23T16:37:12.228 INFO:teuthology.orchestra.run.smithi204.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-23T16:37:12.229 INFO:teuthology.orchestra.run.smithi204.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-23T16:37:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-23T16:37:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-23T16:37:12.233 INFO:teuthology.orchestra.run.smithi204.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-23T16:37:12.246 INFO:teuthology.orchestra.run.smithi079.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-23T16:37:12.260 INFO:teuthology.orchestra.run.smithi204.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-23T16:37:12.266 INFO:teuthology.orchestra.run.smithi204.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-23T16:37:12.267 INFO:teuthology.orchestra.run.smithi204.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-23T16:37:12.294 INFO:teuthology.orchestra.run.smithi204.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3211-g5d349943-1jammy [3,223 kB] 2024-04-23T16:37:12.301 INFO:teuthology.orchestra.run.smithi204.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-23T16:37:12.301 INFO:teuthology.orchestra.run.smithi204.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-23T16:37:12.302 INFO:teuthology.orchestra.run.smithi204.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-23T16:37:12.302 INFO:teuthology.orchestra.run.smithi204.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-23T16:37:12.303 INFO:teuthology.orchestra.run.smithi204.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-23T16:37:12.337 INFO:teuthology.orchestra.run.smithi204.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-23T16:37:12.337 INFO:teuthology.orchestra.run.smithi079.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3211-g5d349943-1jammy [712 kB] 2024-04-23T16:37:12.338 INFO:teuthology.orchestra.run.smithi204.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-23T16:37:12.338 INFO:teuthology.orchestra.run.smithi204.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-23T16:37:12.342 INFO:teuthology.orchestra.run.smithi204.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-23T16:37:12.372 INFO:teuthology.orchestra.run.smithi204.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-23T16:37:12.374 INFO:teuthology.orchestra.run.smithi204.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-23T16:37:12.375 INFO:teuthology.orchestra.run.smithi204.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-23T16:37:12.380 INFO:teuthology.orchestra.run.smithi204.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-23T16:37:12.384 INFO:teuthology.orchestra.run.smithi204.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-23T16:37:12.386 INFO:teuthology.orchestra.run.smithi204.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-23T16:37:12.386 INFO:teuthology.orchestra.run.smithi079.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-23T16:37:12.388 INFO:teuthology.orchestra.run.smithi079.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-23T16:37:12.390 INFO:teuthology.orchestra.run.smithi079.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-23T16:37:12.391 INFO:teuthology.orchestra.run.smithi079.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-23T16:37:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-23T16:37:12.401 INFO:teuthology.orchestra.run.smithi204.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-23T16:37:12.408 INFO:teuthology.orchestra.run.smithi204.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-23T16:37:12.409 INFO:teuthology.orchestra.run.smithi204.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-23T16:37:12.413 INFO:teuthology.orchestra.run.smithi204.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-23T16:37:12.421 INFO:teuthology.orchestra.run.smithi079.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-23T16:37:12.423 INFO:teuthology.orchestra.run.smithi079.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-23T16:37:12.424 INFO:teuthology.orchestra.run.smithi079.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-23T16:37:12.435 INFO:teuthology.orchestra.run.smithi079.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3211-g5d349943-1jammy [357 kB] 2024-04-23T16:37:12.443 INFO:teuthology.orchestra.run.smithi079.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3211-g5d349943-1jammy [32.8 kB] 2024-04-23T16:37:12.443 INFO:teuthology.orchestra.run.smithi204.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-23T16:37:12.444 INFO:teuthology.orchestra.run.smithi079.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3211-g5d349943-1jammy [180 kB] 2024-04-23T16:37:12.445 INFO:teuthology.orchestra.run.smithi204.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-23T16:37:12.445 INFO:teuthology.orchestra.run.smithi204.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-23T16:37:12.448 INFO:teuthology.orchestra.run.smithi079.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3211-g5d349943-1jammy [66.9 kB] 2024-04-23T16:37:12.448 INFO:teuthology.orchestra.run.smithi204.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-23T16:37:12.450 INFO:teuthology.orchestra.run.smithi079.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3211-g5d349943-1jammy [330 kB] 2024-04-23T16:37:12.450 INFO:teuthology.orchestra.run.smithi204.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-23T16:37:12.462 INFO:teuthology.orchestra.run.smithi079.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-23T16:37:12.462 INFO:teuthology.orchestra.run.smithi079.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-23T16:37:12.463 INFO:teuthology.orchestra.run.smithi204.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-23T16:37:12.468 INFO:teuthology.orchestra.run.smithi079.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-23T16:37:12.469 INFO:teuthology.orchestra.run.smithi079.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-23T16:37:12.469 INFO:teuthology.orchestra.run.smithi079.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-23T16:37:12.470 INFO:teuthology.orchestra.run.smithi079.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-23T16:37:12.470 INFO:teuthology.orchestra.run.smithi079.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-23T16:37:12.472 INFO:teuthology.orchestra.run.smithi079.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-23T16:37:12.478 INFO:teuthology.orchestra.run.smithi079.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-23T16:37:12.481 INFO:teuthology.orchestra.run.smithi204.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-23T16:37:12.482 INFO:teuthology.orchestra.run.smithi079.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3211-g5d349943-1jammy [6,894 kB] 2024-04-23T16:37:12.482 INFO:teuthology.orchestra.run.smithi204.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-23T16:37:12.486 INFO:teuthology.orchestra.run.smithi204.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-23T16:37:12.487 INFO:teuthology.orchestra.run.smithi204.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-23T16:37:12.515 INFO:teuthology.orchestra.run.smithi079.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-23T16:37:12.517 INFO:teuthology.orchestra.run.smithi079.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-23T16:37:12.519 INFO:teuthology.orchestra.run.smithi079.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-23T16:37:12.555 INFO:teuthology.orchestra.run.smithi079.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-23T16:37:12.560 INFO:teuthology.orchestra.run.smithi079.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-23T16:37:12.561 INFO:teuthology.orchestra.run.smithi204.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3211-g5d349943-1jammy [712 kB] 2024-04-23T16:37:12.562 INFO:teuthology.orchestra.run.smithi079.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-23T16:37:12.564 INFO:teuthology.orchestra.run.smithi079.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-23T16:37:12.574 INFO:teuthology.orchestra.run.smithi079.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-23T16:37:12.575 INFO:teuthology.orchestra.run.smithi079.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-23T16:37:12.583 INFO:teuthology.orchestra.run.smithi079.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-23T16:37:12.601 INFO:teuthology.orchestra.run.smithi079.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-23T16:37:12.603 INFO:teuthology.orchestra.run.smithi079.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-23T16:37:12.603 INFO:teuthology.orchestra.run.smithi079.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-23T16:37:12.622 INFO:teuthology.orchestra.run.smithi204.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3211-g5d349943-1jammy [357 kB] 2024-04-23T16:37:12.635 INFO:teuthology.orchestra.run.smithi204.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3211-g5d349943-1jammy [32.8 kB] 2024-04-23T16:37:12.637 INFO:teuthology.orchestra.run.smithi204.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3211-g5d349943-1jammy [180 kB] 2024-04-23T16:37:12.637 INFO:teuthology.orchestra.run.smithi079.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-23T16:37:12.639 INFO:teuthology.orchestra.run.smithi079.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-23T16:37:12.644 INFO:teuthology.orchestra.run.smithi204.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3211-g5d349943-1jammy [66.9 kB] 2024-04-23T16:37:12.646 INFO:teuthology.orchestra.run.smithi204.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3211-g5d349943-1jammy [330 kB] 2024-04-23T16:37:12.658 INFO:teuthology.orchestra.run.smithi079.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-23T16:37:12.659 INFO:teuthology.orchestra.run.smithi079.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-23T16:37:12.660 INFO:teuthology.orchestra.run.smithi079.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-23T16:37:12.665 INFO:teuthology.orchestra.run.smithi079.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-23T16:37:12.667 INFO:teuthology.orchestra.run.smithi079.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-23T16:37:12.693 INFO:teuthology.orchestra.run.smithi204.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3211-g5d349943-1jammy [6,894 kB] 2024-04-23T16:37:12.844 INFO:teuthology.orchestra.run.smithi079.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3211-g5d349943-1jammy [112 kB] 2024-04-23T16:37:12.847 INFO:teuthology.orchestra.run.smithi079.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3211-g5d349943-1jammy [466 kB] 2024-04-23T16:37:12.863 INFO:teuthology.orchestra.run.smithi079.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3211-g5d349943-1jammy [25.8 MB] 2024-04-23T16:37:13.273 INFO:teuthology.orchestra.run.smithi204.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3211-g5d349943-1jammy [112 kB] 2024-04-23T16:37:13.278 INFO:teuthology.orchestra.run.smithi204.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3211-g5d349943-1jammy [466 kB] 2024-04-23T16:37:13.325 INFO:teuthology.orchestra.run.smithi204.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3211-g5d349943-1jammy [25.8 MB] 2024-04-23T16:37:14.543 INFO:teuthology.orchestra.run.smithi079.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3211-g5d349943-1jammy [5,092 kB] 2024-04-23T16:37:14.768 INFO:teuthology.orchestra.run.smithi079.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3211-g5d349943-1jammy [244 kB] 2024-04-23T16:37:14.775 INFO:teuthology.orchestra.run.smithi079.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3211-g5d349943-1jammy [124 kB] 2024-04-23T16:37:14.794 INFO:teuthology.orchestra.run.smithi079.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3211-g5d349943-1jammy [1,527 kB] 2024-04-23T16:37:14.881 INFO:teuthology.orchestra.run.smithi079.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3211-g5d349943-1jammy [6,258 kB] 2024-04-23T16:37:15.266 INFO:teuthology.orchestra.run.smithi079.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3211-g5d349943-1jammy [22.8 MB] 2024-04-23T16:37:15.664 INFO:teuthology.orchestra.run.smithi204.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3211-g5d349943-1jammy [5,092 kB] 2024-04-23T16:37:15.874 INFO:teuthology.orchestra.run.smithi204.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3211-g5d349943-1jammy [244 kB] 2024-04-23T16:37:15.895 INFO:teuthology.orchestra.run.smithi204.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3211-g5d349943-1jammy [124 kB] 2024-04-23T16:37:15.899 INFO:teuthology.orchestra.run.smithi204.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3211-g5d349943-1jammy [1,527 kB] 2024-04-23T16:37:16.006 INFO:teuthology.orchestra.run.smithi204.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3211-g5d349943-1jammy [6,258 kB] 2024-04-23T16:37:16.438 INFO:teuthology.orchestra.run.smithi204.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3211-g5d349943-1jammy [22.8 MB] 2024-04-23T16:37:16.908 INFO:teuthology.orchestra.run.smithi079.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3211-g5d349943-1jammy [14.0 kB] 2024-04-23T16:37:16.909 INFO:teuthology.orchestra.run.smithi079.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3211-g5d349943-1jammy [870 kB] 2024-04-23T16:37:16.926 INFO:teuthology.orchestra.run.smithi079.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3211-g5d349943-1jammy [2,436 kB] 2024-04-23T16:37:17.042 INFO:teuthology.orchestra.run.smithi079.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3211-g5d349943-1jammy [791 kB] 2024-04-23T16:37:17.099 INFO:teuthology.orchestra.run.smithi079.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3211-g5d349943-1jammy [148 kB] 2024-04-23T16:37:17.104 INFO:teuthology.orchestra.run.smithi079.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3211-g5d349943-1jammy [2,297 kB] 2024-04-23T16:37:17.277 INFO:teuthology.orchestra.run.smithi079.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3211-g5d349943-1jammy [8,625 kB] 2024-04-23T16:37:17.982 INFO:teuthology.orchestra.run.smithi079.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3211-g5d349943-1jammy [14.2 kB] 2024-04-23T16:37:17.983 INFO:teuthology.orchestra.run.smithi079.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3211-g5d349943-1jammy [51.6 MB] 2024-04-23T16:37:18.143 INFO:teuthology.orchestra.run.smithi204.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3211-g5d349943-1jammy [14.0 kB] 2024-04-23T16:37:18.143 INFO:teuthology.orchestra.run.smithi204.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3211-g5d349943-1jammy [870 kB] 2024-04-23T16:37:18.160 INFO:teuthology.orchestra.run.smithi204.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3211-g5d349943-1jammy [2,436 kB] 2024-04-23T16:37:18.288 INFO:teuthology.orchestra.run.smithi204.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3211-g5d349943-1jammy [791 kB] 2024-04-23T16:37:18.353 INFO:teuthology.orchestra.run.smithi204.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3211-g5d349943-1jammy [148 kB] 2024-04-23T16:37:18.357 INFO:teuthology.orchestra.run.smithi204.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3211-g5d349943-1jammy [2,297 kB] 2024-04-23T16:37:18.530 INFO:teuthology.orchestra.run.smithi204.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3211-g5d349943-1jammy [8,625 kB] 2024-04-23T16:37:19.196 INFO:teuthology.orchestra.run.smithi204.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3211-g5d349943-1jammy [14.2 kB] 2024-04-23T16:37:19.196 INFO:teuthology.orchestra.run.smithi204.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3211-g5d349943-1jammy [51.6 MB] 2024-04-23T16:37:21.671 INFO:teuthology.orchestra.run.smithi079.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3211-g5d349943-1jammy [124 kB] 2024-04-23T16:37:21.673 INFO:teuthology.orchestra.run.smithi079.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3211-g5d349943-1jammy [38.9 kB] 2024-04-23T16:37:21.674 INFO:teuthology.orchestra.run.smithi079.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3211-g5d349943-1jammy [13.6 MB] 2024-04-23T16:37:22.311 INFO:teuthology.orchestra.run.smithi079.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3211-g5d349943-1jammy [91.6 kB] 2024-04-23T16:37:22.602 INFO:teuthology.orchestra.run.smithi204.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3211-g5d349943-1jammy [124 kB] 2024-04-23T16:37:22.675 INFO:teuthology.orchestra.run.smithi204.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3211-g5d349943-1jammy [38.9 kB] 2024-04-23T16:37:22.676 INFO:teuthology.orchestra.run.smithi204.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3211-g5d349943-1jammy [13.6 MB] 2024-04-23T16:37:22.889 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 181 MB in 11s (17.1 MB/s) 2024-04-23T16:37:23.009 INFO:teuthology.orchestra.run.smithi204.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3211-g5d349943-1jammy [91.6 kB] 2024-04-23T16:37:23.033 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-23T16:37:23.126 INFO:teuthology.orchestra.run.smithi079.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-23T16:37:23.131 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-23T16:37:23.207 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-23T16:37:23.561 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-23T16:37:23.587 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-23T16:37:23.627 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-23T16:37:23.641 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 181 MB in 11s (16.0 MB/s) 2024-04-23T16:37:23.802 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-23T16:37:23.902 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-23T16:37:23.907 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-23T16:37:23.981 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-23T16:37:24.000 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-23T16:37:24.006 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-23T16:37:24.047 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-23T16:37:24.287 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-23T16:37:24.313 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-23T16:37:24.345 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-23T16:37:24.467 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-23T16:37:24.494 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T16:37:24.542 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:37:24.673 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-23T16:37:24.699 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-23T16:37:24.740 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-23T16:37:24.937 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-23T16:37:24.964 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T16:37:25.004 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:37:25.185 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-23T16:37:25.211 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T16:37:25.260 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:37:25.323 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-23T16:37:25.350 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T16:37:25.390 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:37:25.630 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-23T16:37:25.657 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T16:37:25.688 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:37:25.718 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-23T16:37:25.743 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-23T16:37:25.784 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-23T16:37:26.033 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-23T16:37:26.060 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T16:37:26.100 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:37:26.104 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librados2. 2024-04-23T16:37:26.130 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../007-librados2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:26.170 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librados2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:26.428 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-23T16:37:26.455 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-23T16:37:26.486 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-23T16:37:26.631 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librbd1. 2024-04-23T16:37:26.657 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../008-librbd1_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:26.698 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librbd1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:26.805 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librados2. 2024-04-23T16:37:26.832 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../007-librados2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:26.872 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librados2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:27.317 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librbd1. 2024-04-23T16:37:27.334 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libcephfs2. 2024-04-23T16:37:27.343 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../008-librbd1_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:27.360 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:27.383 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librbd1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:27.401 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libcephfs2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:27.695 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rados. 2024-04-23T16:37:27.710 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:27.753 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rados (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:27.820 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs2. 2024-04-23T16:37:27.846 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:27.886 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:28.031 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-23T16:37:28.046 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:28.081 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-ceph-argparse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:28.155 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rados. 2024-04-23T16:37:28.170 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:28.213 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rados (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:28.358 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cephfs. 2024-04-23T16:37:28.385 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:28.425 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cephfs (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:28.675 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-23T16:37:28.701 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:28.741 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-argparse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:28.744 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-23T16:37:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:28.811 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:28.994 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cephfs. 2024-04-23T16:37:29.009 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:29.052 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cephfs (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:29.148 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-23T16:37:29.174 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-23T16:37:29.214 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-23T16:37:29.338 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-23T16:37:29.365 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:29.405 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:29.542 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-prettytable. 2024-04-23T16:37:29.568 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-23T16:37:29.608 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-23T16:37:29.700 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-23T16:37:29.715 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-23T16:37:29.750 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-23T16:37:29.844 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rbd. 2024-04-23T16:37:29.860 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:29.894 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rbd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:30.061 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-prettytable. 2024-04-23T16:37:30.087 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-23T16:37:30.127 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-23T16:37:30.264 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-23T16:37:30.290 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-23T16:37:30.330 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-23T16:37:30.380 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rbd. 2024-04-23T16:37:30.395 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:30.438 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rbd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:30.608 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-23T16:37:30.623 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-23T16:37:30.666 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-23T16:37:30.783 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-23T16:37:30.810 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-23T16:37:30.850 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-23T16:37:30.969 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-23T16:37:30.996 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-23T16:37:31.036 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-23T16:37:31.127 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-23T16:37:31.154 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-23T16:37:31.219 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-23T16:37:31.347 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua5.1. 2024-04-23T16:37:31.374 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-23T16:37:31.414 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-23T16:37:31.497 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-23T16:37:31.512 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-23T16:37:31.546 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-23T16:37:31.733 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua-any. 2024-04-23T16:37:31.760 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-23T16:37:31.800 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-23T16:37:31.876 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua5.1. 2024-04-23T16:37:31.902 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-23T16:37:31.941 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-23T16:37:32.027 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package zip. 2024-04-23T16:37:32.043 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-23T16:37:32.094 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking zip (3.0-12build2) ... 2024-04-23T16:37:32.253 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua-any. 2024-04-23T16:37:32.269 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-23T16:37:32.311 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-23T16:37:32.406 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package unzip. 2024-04-23T16:37:32.432 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-23T16:37:32.464 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-23T16:37:32.530 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package zip. 2024-04-23T16:37:32.546 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-23T16:37:32.580 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking zip (3.0-12build2) ... 2024-04-23T16:37:32.876 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package unzip. 2024-04-23T16:37:32.876 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package luarocks. 2024-04-23T16:37:32.902 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-23T16:37:32.902 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-23T16:37:32.942 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-23T16:37:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-23T16:37:33.325 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librgw2. 2024-04-23T16:37:33.329 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package luarocks. 2024-04-23T16:37:33.352 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../025-librgw2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:33.356 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-23T16:37:33.396 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-23T16:37:33.425 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librgw2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:33.753 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librgw2. 2024-04-23T16:37:33.769 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../025-librgw2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:33.803 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librgw2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:33.969 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rgw. 2024-04-23T16:37:33.984 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:34.036 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:34.322 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rgw. 2024-04-23T16:37:34.338 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:34.339 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-23T16:37:34.365 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-23T16:37:34.397 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:34.405 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-23T16:37:34.691 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libradosstriper1. 2024-04-23T16:37:34.718 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:34.726 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-23T16:37:34.752 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-23T16:37:34.758 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libradosstriper1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:34.792 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-23T16:37:35.095 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libradosstriper1. 2024-04-23T16:37:35.123 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:35.162 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libradosstriper1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:35.222 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-common. 2024-04-23T16:37:35.247 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:35.289 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:35.515 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-common. 2024-04-23T16:37:35.542 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:35.573 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:36.675 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-base. 2024-04-23T16:37:36.701 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:36.759 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-base (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:36.912 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-base. 2024-04-23T16:37:36.939 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:36.987 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-base (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:37.363 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-23T16:37:37.389 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-23T16:37:37.429 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-23T16:37:37.566 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-23T16:37:37.593 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-23T16:37:37.633 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-23T16:37:37.690 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cheroot. 2024-04-23T16:37:37.716 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-23T16:37:37.748 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cheroot. 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-23T16:37:38.002 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-23T16:37:38.060 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-23T16:37:38.086 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-23T16:37:38.126 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-23T16:37:38.322 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-23T16:37:38.348 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-23T16:37:38.389 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-23T16:37:38.429 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-23T16:37:38.455 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-23T16:37:38.496 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-23T16:37:38.773 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-23T16:37:38.789 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-23T16:37:38.823 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-23T16:37:38.850 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-23T16:37:38.877 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-23T16:37:38.916 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-23T16:37:39.068 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-tempora. 2024-04-23T16:37:39.094 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-23T16:37:39.134 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-23T16:37:39.219 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-23T16:37:39.246 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-23T16:37:39.286 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-23T16:37:39.446 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-portend. 2024-04-23T16:37:39.472 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-23T16:37:39.512 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-23T16:37:39.547 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempora. 2024-04-23T16:37:39.563 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-23T16:37:39.597 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-23T16:37:39.815 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-23T16:37:39.841 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-23T16:37:39.882 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-23T16:37:39.885 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-portend. 2024-04-23T16:37:39.910 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-23T16:37:39.958 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-23T16:37:40.193 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-23T16:37:40.219 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-23T16:37:40.259 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-23T16:37:40.261 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-23T16:37:40.288 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-23T16:37:40.327 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-23T16:37:40.622 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-natsort. 2024-04-23T16:37:40.623 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-23T16:37:40.648 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-23T16:37:40.649 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-23T16:37:40.688 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-23T16:37:40.688 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-23T16:37:40.991 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-logutils. 2024-04-23T16:37:41.018 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-23T16:37:41.026 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-natsort. 2024-04-23T16:37:41.053 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-23T16:37:41.057 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-23T16:37:41.092 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-23T16:37:41.344 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-23T16:37:41.371 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-23T16:37:41.395 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-logutils. 2024-04-23T16:37:41.410 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-23T16:37:41.422 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-23T16:37:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-23T16:37:41.747 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-mako. 2024-04-23T16:37:41.765 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-23T16:37:41.773 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-23T16:37:41.791 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-23T16:37:41.821 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-23T16:37:41.831 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-23T16:37:42.134 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-mako. 2024-04-23T16:37:42.141 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-23T16:37:42.161 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-23T16:37:42.168 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-23T16:37:42.201 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-23T16:37:42.208 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-23T16:37:42.495 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-23T16:37:42.502 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-23T16:37:42.511 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-23T16:37:42.529 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-23T16:37:42.545 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-23T16:37:42.569 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-23T16:37:42.857 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-23T16:37:42.883 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-23T16:37:42.923 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-23T16:37:43.156 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-webob. 2024-04-23T16:37:43.182 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-23T16:37:43.209 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webob. 2024-04-23T16:37:43.237 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-23T16:37:43.276 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-23T16:37:43.646 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-waitress. 2024-04-23T16:37:43.647 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-23T16:37:43.673 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-23T16:37:43.715 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-23T16:37:44.023 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempita. 2024-04-23T16:37:44.050 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-23T16:37:44.050 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-waitress. 2024-04-23T16:37:44.076 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-23T16:37:44.090 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-23T16:37:44.120 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-23T16:37:44.385 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-paste. 2024-04-23T16:37:44.411 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-23T16:37:44.420 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-tempita. 2024-04-23T16:37:44.447 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-23T16:37:44.451 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-23T16:37:44.486 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-23T16:37:44.772 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-paste. 2024-04-23T16:37:44.799 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-23T16:37:44.839 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-23T16:37:45.030 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-23T16:37:45.057 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-23T16:37:45.192 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-23T16:37:45.219 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-23T16:37:45.259 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-23T16:37:45.488 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-23T16:37:45.553 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-23T16:37:45.580 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-23T16:37:45.620 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-23T16:37:45.883 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-23T16:37:45.906 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-webtest. 2024-04-23T16:37:45.909 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-23T16:37:45.922 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-23T16:37:45.949 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-23T16:37:45.956 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-23T16:37:46.226 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pecan. 2024-04-23T16:37:46.252 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-23T16:37:46.261 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webtest. 2024-04-23T16:37:46.284 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-23T16:37:46.287 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-23T16:37:46.327 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-23T16:37:46.605 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-certifi. 2024-04-23T16:37:46.630 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pecan. 2024-04-23T16:37:46.632 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-23T16:37:46.656 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-23T16:37:46.663 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-23T16:37:46.688 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-23T16:37:46.991 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-idna. 2024-04-23T16:37:47.010 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-certifi. 2024-04-23T16:37:47.018 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-23T16:37:47.025 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-23T16:37:47.058 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-23T16:37:47.060 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-23T16:37:47.369 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-urllib3. 2024-04-23T16:37:47.379 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-idna. 2024-04-23T16:37:47.396 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-23T16:37:47.406 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-23T16:37:47.436 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-23T16:37:47.446 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-23T16:37:47.756 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-requests. 2024-04-23T16:37:47.757 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-urllib3. 2024-04-23T16:37:47.782 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-23T16:37:47.784 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-23T16:37:47.822 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-23T16:37:47.823 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-23T16:37:48.136 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests. 2024-04-23T16:37:48.158 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-23T16:37:48.162 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-23T16:37:48.185 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-23T16:37:48.202 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-23T16:37:48.225 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-23T16:37:48.530 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-23T16:37:48.536 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-23T16:37:48.552 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:48.557 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-23T16:37:48.594 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:48.596 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-23T16:37:48.899 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-23T16:37:48.915 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:48.949 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:48.974 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-23T16:37:49.001 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:49.041 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:49.335 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-23T16:37:49.361 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-23T16:37:49.402 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-23T16:37:49.504 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-23T16:37:49.532 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:49.571 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:49.665 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-openssl. 2024-04-23T16:37:49.680 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-23T16:37:49.715 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-23T16:37:49.890 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-23T16:37:49.917 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-23T16:37:49.957 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-23T16:37:50.085 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr. 2024-04-23T16:37:50.114 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:50.151 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:50.243 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-openssl. 2024-04-23T16:37:50.270 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-23T16:37:50.318 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-23T16:37:50.564 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mon. 2024-04-23T16:37:50.590 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:50.630 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mon (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:50.654 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr. 2024-04-23T16:37:50.681 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:50.721 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:51.116 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mon. 2024-04-23T16:37:51.143 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:51.175 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-osd. 2024-04-23T16:37:51.182 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mon (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:51.191 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:51.234 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-osd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:51.711 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-osd. 2024-04-23T16:37:51.738 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:51.769 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-osd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:52.312 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph. 2024-04-23T16:37:52.327 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../066-ceph_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:52.371 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:52.814 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph. 2024-04-23T16:37:52.830 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../066-ceph_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:52.872 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:52.940 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-fuse. 2024-04-23T16:37:52.967 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:53.007 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:53.200 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-fuse. 2024-04-23T16:37:53.227 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:53.266 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:53.365 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mds. 2024-04-23T16:37:53.391 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:53.435 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mds (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:53.622 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mds. 2024-04-23T16:37:53.649 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:53.688 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mds (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:53.839 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package cephadm. 2024-04-23T16:37:53.865 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../069-cephadm_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:53.913 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:54.091 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephadm. 2024-04-23T16:37:54.118 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../069-cephadm_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:37:54.158 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:54.233 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-23T16:37:54.259 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-23T16:37:54.299 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-23T16:37:54.530 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-23T16:37:54.557 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-23T16:37:54.627 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-23T16:37:54.642 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-23T16:37:54.644 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-23T16:37:54.685 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-23T16:37:54.938 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-23T16:37:54.953 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-23T16:37:54.997 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-23T16:37:55.750 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-babel. 2024-04-23T16:37:55.777 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-23T16:37:55.816 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-23T16:37:56.077 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-babel. 2024-04-23T16:37:56.104 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-23T16:37:56.136 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jinja2. 2024-04-23T16:37:56.143 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-23T16:37:56.163 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-23T16:37:56.202 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-23T16:37:56.455 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jinja2. 2024-04-23T16:37:56.482 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-23T16:37:56.521 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-23T16:37:56.564 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-23T16:37:56.592 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:56.631 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:56.849 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-23T16:37:56.876 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:56.907 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:56.984 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-23T16:37:57.011 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-23T16:37:57.051 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-23T16:37:57.228 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-23T16:37:57.243 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-23T16:37:57.277 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-23T16:37:57.354 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-routes. 2024-04-23T16:37:57.381 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-23T16:37:57.420 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-23T16:37:57.556 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-routes. 2024-04-23T16:37:57.582 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-23T16:37:57.622 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-23T16:37:57.774 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-23T16:37:57.801 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:57.841 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:57.925 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-23T16:37:57.952 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:37:57.992 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:37:59.231 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-23T16:37:59.247 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-23T16:37:59.263 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-23T16:37:59.279 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-23T16:37:59.289 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-23T16:37:59.321 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-23T16:37:59.775 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-joblib. 2024-04-23T16:37:59.776 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-joblib. 2024-04-23T16:37:59.791 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-23T16:37:59.803 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-23T16:37:59.825 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-23T16:37:59.842 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-23T16:38:00.205 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-23T16:38:00.232 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-23T16:38:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-23T16:38:00.265 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-23T16:38:00.271 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-23T16:38:00.303 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-23T16:38:00.591 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn. 2024-04-23T16:38:00.607 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-sklearn. 2024-04-23T16:38:00.618 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-23T16:38:00.634 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-23T16:38:00.657 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-23T16:38:00.673 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-23T16:38:01.341 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-23T16:38:01.368 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:38:01.374 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-23T16:38:01.402 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:38:01.485 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:01.527 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:02.480 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cachetools. 2024-04-23T16:38:02.489 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cachetools. 2024-04-23T16:38:02.507 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-23T16:38:02.516 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-23T16:38:02.547 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-23T16:38:02.555 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-23T16:38:02.841 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-23T16:38:02.858 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-23T16:38:02.868 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-23T16:38:02.885 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-23T16:38:02.908 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-23T16:38:02.925 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-23T16:38:03.186 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-23T16:38:03.202 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-23T16:38:03.236 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-23T16:38:03.245 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-23T16:38:03.273 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-23T16:38:03.312 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-23T16:38:03.523 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rsa. 2024-04-23T16:38:03.550 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-23T16:38:03.589 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-23T16:38:03.624 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rsa. 2024-04-23T16:38:03.651 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-23T16:38:03.690 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-23T16:38:03.910 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-google-auth. 2024-04-23T16:38:03.937 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-23T16:38:03.976 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-23T16:38:04.036 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-google-auth. 2024-04-23T16:38:04.063 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-23T16:38:04.102 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-23T16:38:04.455 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-23T16:38:04.482 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-23T16:38:04.521 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-23T16:38:04.573 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-23T16:38:04.600 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-23T16:38:04.640 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-23T16:38:04.917 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-websocket. 2024-04-23T16:38:04.945 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-23T16:38:04.984 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-23T16:38:04.991 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-websocket. 2024-04-23T16:38:05.019 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-23T16:38:05.057 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-23T16:38:05.304 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-23T16:38:05.331 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-23T16:38:05.379 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-23T16:38:05.403 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-23T16:38:05.430 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-23T16:38:05.469 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-23T16:38:06.036 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-23T16:38:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:38:06.075 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-23T16:38:06.093 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:06.103 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:38:06.141 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:06.397 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-23T16:38:06.424 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-23T16:38:06.461 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-23T16:38:06.463 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-23T16:38:06.488 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-23T16:38:06.527 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-23T16:38:06.833 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-23T16:38:06.861 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-23T16:38:06.873 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-23T16:38:06.900 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-23T16:38:06.900 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-23T16:38:06.939 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-23T16:38:07.150 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package jq. 2024-04-23T16:38:07.161 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package jq. 2024-04-23T16:38:07.166 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-23T16:38:07.177 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-23T16:38:07.208 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-23T16:38:07.211 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-23T16:38:07.439 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package socat. 2024-04-23T16:38:07.461 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package socat. 2024-04-23T16:38:07.466 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-23T16:38:07.488 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-23T16:38:07.505 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-23T16:38:07.528 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-23T16:38:07.826 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package xmlstarlet. 2024-04-23T16:38:07.839 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package xmlstarlet. 2024-04-23T16:38:07.853 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-23T16:38:07.866 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-23T16:38:07.894 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-23T16:38:07.906 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-23T16:38:08.247 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-test. 2024-04-23T16:38:08.269 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-test. 2024-04-23T16:38:08.275 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:08.284 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:08.313 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-test (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:08.343 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-test (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:10.607 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-volume. 2024-04-23T16:38:10.634 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:38:10.643 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-volume. 2024-04-23T16:38:10.670 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T16:38:10.673 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-volume (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:10.709 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-volume (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:10.952 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-23T16:38:10.980 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:11.019 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs-dev (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:11.022 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-23T16:38:11.049 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:11.089 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libcephfs-dev (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:11.338 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-23T16:38:11.366 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-23T16:38:11.383 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-23T16:38:11.410 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-23T16:38:11.421 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-23T16:38:11.450 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-23T16:38:11.751 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-23T16:38:11.763 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-23T16:38:11.779 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-23T16:38:11.790 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-23T16:38:11.818 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-23T16:38:11.829 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-23T16:38:12.139 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package nvme-cli. 2024-04-23T16:38:12.159 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package nvme-cli. 2024-04-23T16:38:12.166 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-23T16:38:12.186 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-23T16:38:12.197 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-23T16:38:12.225 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-23T16:38:12.586 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package pkg-config. 2024-04-23T16:38:12.606 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package pkg-config. 2024-04-23T16:38:12.613 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-23T16:38:12.622 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-23T16:38:12.652 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-23T16:38:12.681 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-23T16:38:12.922 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-23T16:38:12.938 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-23T16:38:12.942 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-23T16:38:12.967 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-23T16:38:12.980 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-23T16:38:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-23T16:38:13.408 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-23T16:38:13.436 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-23T16:38:13.446 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-23T16:38:13.474 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-23T16:38:13.475 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-23T16:38:13.512 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-23T16:38:13.778 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastescript. 2024-04-23T16:38:13.799 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pastescript. 2024-04-23T16:38:13.805 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-23T16:38:13.826 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-23T16:38:13.844 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-23T16:38:13.857 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-23T16:38:14.127 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pluggy. 2024-04-23T16:38:14.155 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-23T16:38:14.165 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pluggy. 2024-04-23T16:38:14.192 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-23T16:38:14.193 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-23T16:38:14.231 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-23T16:38:14.506 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-psutil. 2024-04-23T16:38:14.533 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-23T16:38:14.534 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-psutil. 2024-04-23T16:38:14.562 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-23T16:38:14.572 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-23T16:38:14.592 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-23T16:38:14.909 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-py. 2024-04-23T16:38:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-23T16:38:14.937 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-py. 2024-04-23T16:38:14.964 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-23T16:38:14.983 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-23T16:38:15.003 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-23T16:38:15.304 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pygments. 2024-04-23T16:38:15.331 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-23T16:38:15.340 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pygments. 2024-04-23T16:38:15.368 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-23T16:38:15.370 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-23T16:38:15.407 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-23T16:38:15.820 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-23T16:38:15.833 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-23T16:38:15.847 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-23T16:38:15.861 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-23T16:38:15.879 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-23T16:38:15.900 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-23T16:38:16.164 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-toml. 2024-04-23T16:38:16.169 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-toml. 2024-04-23T16:38:16.185 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-23T16:38:16.191 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-23T16:38:16.219 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-23T16:38:16.230 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-23T16:38:16.522 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pytest. 2024-04-23T16:38:16.533 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pytest. 2024-04-23T16:38:16.550 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-23T16:38:16.561 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-23T16:38:16.588 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-23T16:38:16.600 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-23T16:38:16.903 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplejson. 2024-04-23T16:38:16.919 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-23T16:38:16.926 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-simplejson. 2024-04-23T16:38:16.953 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-23T16:38:16.953 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-23T16:38:16.984 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-23T16:38:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-23T16:38:17.211 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-23T16:38:17.228 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-23T16:38:17.244 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-23T16:38:17.247 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-23T16:38:17.286 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-23T16:38:17.942 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package radosgw. 2024-04-23T16:38:17.969 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../116-radosgw_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:17.993 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package radosgw. 2024-04-23T16:38:17.999 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking radosgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:18.021 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../116-radosgw_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:18.060 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking radosgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:18.794 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package rbd-fuse. 2024-04-23T16:38:18.822 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:18.861 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking rbd-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:18.871 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-fuse. 2024-04-23T16:38:18.888 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T16:38:18.929 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:19.157 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-23T16:38:19.234 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-23T16:38:19.408 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-23T16:38:19.493 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-23T16:38:19.534 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-23T16:38:19.611 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-23T16:38:19.798 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-23T16:38:19.853 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-23T16:38:20.143 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-23T16:38:20.226 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-23T16:38:20.566 INFO:teuthology.orchestra.run.smithi079.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-23T16:38:20.644 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:20.669 INFO:teuthology.orchestra.run.smithi204.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-23T16:38:20.755 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:21.136 INFO:teuthology.orchestra.run.smithi079.stdout:Adding system user cephadm....done 2024-04-23T16:38:21.293 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-23T16:38:21.304 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user cephadm....done 2024-04-23T16:38:21.428 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-23T16:38:21.577 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-23T16:38:21.679 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-23T16:38:21.828 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-23T16:38:21.923 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-23T16:38:21.938 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-23T16:38:22.049 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-23T16:38:22.181 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-23T16:38:22.308 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-23T16:38:22.432 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-23T16:38:22.559 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-23T16:38:22.568 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-23T16:38:22.694 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-23T16:38:22.885 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-23T16:38:22.979 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-23T16:38:23.238 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-23T16:38:23.331 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-23T16:38:23.514 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-23T16:38:23.582 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-23T16:38:23.782 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-23T16:38:23.858 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-23T16:38:24.042 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-23T16:38:24.185 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-23T16:38:24.310 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-23T16:38:24.419 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-23T16:38:24.586 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-23T16:38:24.662 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-23T16:38:24.911 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-ceph-argparse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:24.913 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-argparse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:25.247 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-23T16:38:25.290 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-23T16:38:25.373 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-23T16:38:25.415 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-23T16:38:25.499 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-23T16:38:25.541 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-23T16:38:25.625 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-23T16:38:25.667 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-23T16:38:25.884 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-23T16:38:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-23T16:38:26.272 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-23T16:38:26.307 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-23T16:38:26.371 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-23T16:38:26.372 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-23T16:38:26.408 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-23T16:38:26.489 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-23T16:38:26.489 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-23T16:38:26.615 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-23T16:38:26.848 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-23T16:38:26.975 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-23T16:38:27.016 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-23T16:38:27.142 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-23T16:38:27.243 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-23T16:38:27.410 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-23T16:38:27.502 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-23T16:38:27.652 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-23T16:38:27.795 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up zip (3.0-12build2) ... 2024-04-23T16:38:27.897 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-23T16:38:27.929 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up zip (3.0-12build2) ... 2024-04-23T16:38:28.055 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-23T16:38:28.598 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-23T16:38:28.715 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-23T16:38:28.858 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-23T16:38:28.975 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-23T16:38:28.984 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-23T16:38:29.101 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-23T16:38:29.110 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-23T16:38:29.210 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-23T16:38:29.411 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-23T16:38:29.495 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-23T16:38:29.871 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-23T16:38:30.122 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-23T16:38:30.139 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-23T16:38:30.381 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-23T16:38:30.482 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-23T16:38:30.749 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-23T16:38:30.817 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-23T16:38:31.034 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-23T16:38:31.160 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-23T16:38:31.377 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-23T16:38:31.386 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-23T16:38:31.503 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:31.611 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-23T16:38:31.729 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:31.797 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-idna (3.3-1) ... 2024-04-23T16:38:32.022 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-idna (3.3-1) ... 2024-04-23T16:38:32.115 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-23T16:38:32.340 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-23T16:38:33.295 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-23T16:38:33.538 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-23T16:38:33.547 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:38:33.681 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-23T16:38:33.765 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:38:33.899 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-23T16:38:33.949 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-23T16:38:34.075 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-23T16:38:34.167 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-23T16:38:34.201 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-23T16:38:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-23T16:38:34.420 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-23T16:38:34.478 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-23T16:38:34.715 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-23T16:38:34.721 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-23T16:38:34.966 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-23T16:38:34.972 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:38:35.098 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-23T16:38:35.217 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:38:35.343 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-23T16:38:35.399 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-23T16:38:35.628 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-23T16:38:35.650 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-23T16:38:35.862 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-23T16:38:35.910 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-23T16:38:36.122 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-23T16:38:36.153 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-23T16:38:36.365 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-23T16:38:36.429 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-23T16:38:36.633 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-23T16:38:36.682 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-23T16:38:36.808 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-23T16:38:36.900 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-23T16:38:37.018 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-23T16:38:37.109 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-23T16:38:37.218 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-23T16:38:37.278 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-23T16:38:37.420 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-23T16:38:37.495 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-23T16:38:37.697 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-23T16:38:37.779 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-23T16:38:38.030 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-23T16:38:38.032 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-23T16:38:38.156 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-23T16:38:38.291 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-23T16:38:38.300 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-23T16:38:38.380 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-23T16:38:38.426 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-23T16:38:38.552 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-23T16:38:38.625 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-23T16:38:38.634 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-23T16:38:38.752 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-23T16:38:38.870 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-23T16:38:38.988 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-23T16:38:39.120 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-23T16:38:39.357 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-23T16:38:39.380 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-23T16:38:39.506 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-23T16:38:39.616 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-23T16:38:39.742 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-23T16:38:39.765 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:38:39.891 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-23T16:38:40.001 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T16:38:40.102 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-23T16:38:40.209 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-23T16:38:40.395 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-23T16:38:40.502 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-23T16:38:40.689 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-23T16:38:40.762 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-23T16:38:40.887 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-23T16:38:40.948 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-23T16:38:41.074 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-23T16:38:41.138 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-23T16:38:41.342 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-23T16:38:41.398 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-23T16:38:41.610 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-23T16:38:41.760 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-23T16:38:41.956 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-23T16:38:42.145 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-23T16:38:42.422 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-23T16:38:42.626 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-23T16:38:42.775 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-23T16:38:42.901 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librados2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:42.970 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-23T16:38:43.027 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.088 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librados2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.153 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-23T16:38:43.222 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.365 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-23T16:38:43.464 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.573 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libradosstriper1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.700 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-23T16:38:43.702 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libcephfs2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.811 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libradosstriper1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:43.912 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-23T16:38:43.976 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librbd1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:44.086 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-modules-core (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:44.181 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librbd1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:44.195 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:44.307 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-modules-core (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-23T16:38:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-23T16:38:44.433 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:44.644 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-23T16:38:44.644 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-23T16:38:44.988 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs-dev (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:45.093 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libcephfs-dev (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:45.097 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rados (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:45.194 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rados (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:45.207 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librgw2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:45.295 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librgw2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:45.341 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-23T16:38:45.413 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-23T16:38:46.621 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rbd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:46.684 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rbd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:46.746 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:46.810 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up rbd-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:46.872 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:46.936 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:46.998 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cephfs (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:47.053 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cephfs (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:47.125 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:47.154 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:47.651 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group ceph....done 2024-04-23T16:38:47.656 INFO:teuthology.orchestra.run.smithi079.stdout:Adding group ceph....done 2024-04-23T16:38:48.010 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user ceph....done 2024-04-23T16:38:48.033 INFO:teuthology.orchestra.run.smithi079.stdout:Adding system user ceph....done 2024-04-23T16:38:48.169 INFO:teuthology.orchestra.run.smithi079.stdout:Setting system user ceph properties....done 2024-04-23T16:38:48.179 INFO:teuthology.orchestra.run.smithi204.stdout:Setting system user ceph properties....done 2024-04-23T16:38:48.184 INFO:teuthology.orchestra.run.smithi079.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-23T16:38:48.191 INFO:teuthology.orchestra.run.smithi204.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-23T16:38:48.320 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-23T16:38:48.328 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-23T16:38:48.630 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-23T16:38:48.664 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-23T16:38:49.085 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-test (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:49.123 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-test (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:49.211 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up radosgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:49.249 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up radosgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:49.678 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-23T16:38:49.678 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-23T16:38:49.699 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-23T16:38:49.699 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-23T16:38:50.146 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-base (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:50.218 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-base (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:50.530 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-23T16:38:50.559 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-23T16:38:51.038 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mds (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:51.243 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mds (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:51.367 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-23T16:38:51.367 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-23T16:38:51.582 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-23T16:38:51.583 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-23T16:38:51.842 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:52.051 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:52.079 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-23T16:38:52.079 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-23T16:38:52.258 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-23T16:38:52.259 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-23T16:38:52.568 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-osd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:52.703 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-osd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:52.847 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-23T16:38:52.847 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-23T16:38:52.981 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-23T16:38:52.981 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-23T16:38:53.295 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:53.412 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:53.538 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:53.539 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mon (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:53.698 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mon (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:53.751 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-23T16:38:53.751 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-23T16:38:53.910 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-23T16:38:53.910 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-23T16:38:54.190 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.333 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.366 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.434 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-dashboard (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.484 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.560 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-volume (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.593 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-dashboard (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.769 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-volume (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T16:38:54.897 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-23T16:38:54.998 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-23T16:38:55.134 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-23T16:38:55.139 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-23T16:38:55.248 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-23T16:38:55.401 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-23T16:38:55.944 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-23T16:38:56.494 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-23T16:38:56.581 INFO:teuthology.orchestra.run.smithi079.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-23T16:38:56.587 DEBUG:teuthology.parallel:result is None 2024-04-23T16:38:57.134 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-23T16:38:57.139 DEBUG:teuthology.parallel:result is None 2024-04-23T16:38:57.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:38:57.295 DEBUG:teuthology.orchestra.run.smithi079:> dpkg-query -W -f '${Version}' ceph 2024-04-23T16:38:57.314 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-3211-g5d349943-1jammy 2024-04-23T16:38:57.315 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3211-g5d349943-1jammy 2024-04-23T16:38:57.315 INFO:teuthology.task.install:The correct ceph version 19.0.0-3211-g5d349943-1jammy is installed. 2024-04-23T16:38:57.316 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:38:57.538 DEBUG:teuthology.orchestra.run.smithi204:> dpkg-query -W -f '${Version}' ceph 2024-04-23T16:38:57.557 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-3211-g5d349943-1jammy 2024-04-23T16:38:57.558 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3211-g5d349943-1jammy 2024-04-23T16:38:57.558 INFO:teuthology.task.install:The correct ceph version 19.0.0-3211-g5d349943-1jammy is installed. 2024-04-23T16:38:57.559 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T16:38:57.559 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:38:57.560 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T16:38:57.574 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:38:57.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T16:38:57.613 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T16:38:57.613 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:38:57.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T16:38:57.628 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T16:38:57.684 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:38:57.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T16:38:57.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T16:38:57.750 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T16:38:57.750 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:38:57.750 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T16:38:57.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T16:38:57.816 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:38:57.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T16:38:57.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T16:38:57.882 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T16:38:57.883 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:38:57.883 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T16:38:57.895 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T16:38:57.947 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:38:57.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T16:38:57.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T16:38:58.015 INFO:teuthology.run_tasks:Running task ceph... 2024-04-23T16:38:58.098 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-23T16:38:58.098 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2024-04-23T16:38:58.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-04-23T16:38:58.114 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-23T16:38:58.114 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T16:38:58.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T16:38:58.170 INFO:tasks.ceph:Creating extra log directories... 2024-04-23T16:38:58.170 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T16:38:58.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T16:38:58.236 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-23T16:38:58.236 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-23T16:38:58.236 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776', 'branch': 'wip-yuri3-testing-2024-04-05-0825', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7669776', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2024-04-05-0825'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2024-04-05-0825', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 3753, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2024-04-05-0825', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'targets': {'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFDP0YXGDWK/8K5PzJgh32pqa0us48n1hYj5PEm81TzEE8Az2ZVDtbqXNkUvfKB1vdy/N7OM4zY+PQYtF04+StQ=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCerRUXaZw0MajyMm8FnD4TNG6jVXDe6qeBA40sLC16SJmuUrgYehUwaKZwbefdN1o8CcKZb6wtjQFzQtsKkufA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_14:14:08', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-23T16:38:58.237 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T16:38:58.269 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T16:38:58.276 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T16:38:58.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T16:38:58.331 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:38:58.331 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T16:38:58.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T16:38:58.377 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-23 16:33:42.596745381 +0000 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-23 16:33:42.320747847 +0000 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-23 16:33:42.320747847 +0000 2024-04-23T16:38:58.425 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-04-23T16:38:58.426 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T16:38:58.479 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-23T16:38:58.479 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-23T16:38:58.479 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000401774 s, 1.3 MB/s 2024-04-23T16:38:58.480 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T16:38:58.530 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-04-23T16:38:58.576 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-23 16:33:43.032741486 +0000 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-23 16:33:42.812743452 +0000 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-23 16:33:42.812743452 +0000 2024-04-23T16:38:58.577 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-04-23T16:38:58.577 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T16:38:58.631 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-23T16:38:58.631 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-23T16:38:58.631 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000361968 s, 1.4 MB/s 2024-04-23T16:38:58.632 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T16:38:58.682 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-04-23T16:38:58.728 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T16:38:58.728 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:58.728 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-23T16:38:58.728 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:58.729 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-23 16:33:43.536736985 +0000 2024-04-23T16:38:58.729 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-23 16:33:43.280739271 +0000 2024-04-23T16:38:58.729 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-23 16:33:43.280739271 +0000 2024-04-23T16:38:58.729 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-04-23T16:38:58.729 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T16:38:58.782 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-23T16:38:58.782 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-23T16:38:58.782 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000429586 s, 1.2 MB/s 2024-04-23T16:38:58.784 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T16:38:58.834 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-23 16:33:44.016732698 +0000 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-23 16:33:43.772734877 +0000 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-23 16:33:43.772734877 +0000 2024-04-23T16:38:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-04-23T16:38:58.887 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T16:38:58.939 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-23T16:38:58.939 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-23T16:38:58.939 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000392987 s, 1.3 MB/s 2024-04-23T16:38:58.940 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T16:38:58.990 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-23T16:38:58.990 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:38:58.990 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T16:38:58.996 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T16:38:58.996 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-04-23T16:38:59.044 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T16:38:59.044 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:59.044 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-04-23T16:38:59.045 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:59.045 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 16:33:43.852808790 +0000 2024-04-23T16:38:59.045 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 16:33:43.600811042 +0000 2024-04-23T16:38:59.045 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 16:33:43.600811042 +0000 2024-04-23T16:38:59.045 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-04-23T16:38:59.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T16:38:59.098 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T16:38:59.098 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T16:38:59.098 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000446486 s, 1.1 MB/s 2024-04-23T16:38:59.099 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T16:38:59.149 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-04-23T16:38:59.195 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T16:38:59.195 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:59.195 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-23T16:38:59.195 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:59.195 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 16:33:44.336804468 +0000 2024-04-23T16:38:59.195 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 16:33:44.072806825 +0000 2024-04-23T16:38:59.196 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 16:33:44.072806825 +0000 2024-04-23T16:38:59.196 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-04-23T16:38:59.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T16:38:59.249 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T16:38:59.249 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T16:38:59.250 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000341284 s, 1.5 MB/s 2024-04-23T16:38:59.251 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T16:38:59.301 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-04-23T16:38:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 16:33:44.896799466 +0000 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 16:33:44.608802038 +0000 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 16:33:44.608802038 +0000 2024-04-23T16:38:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-04-23T16:38:59.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T16:38:59.403 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T16:38:59.403 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T16:38:59.403 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000415464 s, 1.2 MB/s 2024-04-23T16:38:59.404 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T16:38:59.452 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 16:33:45.396794999 +0000 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 16:33:45.140797287 +0000 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 16:33:45.140797287 +0000 2024-04-23T16:38:59.500 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-04-23T16:38:59.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T16:38:59.554 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T16:38:59.554 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T16:38:59.554 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000333818 s, 1.5 MB/s 2024-04-23T16:38:59.555 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T16:38:59.605 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-23T16:38:59.605 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-23T16:38:59.605 INFO:tasks.ceph:Generating config... 2024-04-23T16:38:59.606 INFO:tasks.ceph:[client] debug monc = 20 2024-04-23T16:38:59.606 INFO:tasks.ceph:[client] debug ms = 1 2024-04-23T16:38:59.606 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-23T16:38:59.606 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-23T16:38:59.606 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-04-23T16:38:59.606 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-04-23T16:38:59.606 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-23T16:38:59.606 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-23T16:38:59.607 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-04-23T16:38:59.607 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-23T16:38:59.607 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-23T16:38:59.607 INFO:tasks.ceph:[global] ms type = async 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-23T16:38:59.607 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-23T16:38:59.608 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-23T16:38:59.608 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-23T16:38:59.608 INFO:tasks.ceph:[mon] paxos min = 10 2024-04-23T16:38:59.608 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-23T16:38:59.608 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-23T16:38:59.608 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-23T16:38:59.609 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-23T16:38:59.609 INFO:tasks.ceph:Setting up mon.a... 2024-04-23T16:38:59.609 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-23T16:38:59.642 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.keyring 2024-04-23T16:38:59.664 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-23T16:38:59.742 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T16:38:59.796 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.79'), ('mon.c', '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]'), ('mon.b', '172.21.15.204')] 2024-04-23T16:38:59.796 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.79,[v2:172.21.15.79:3301,v1:172.21.15.79:6790],172.21.15.204', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-23T16:38:59.797 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:38:59.797 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T16:38:59.844 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.79 --addv c '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]' --add b 172.21.15.204 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:38:59.931 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:38:59.931 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: generated fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = quincy 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 0 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:min_mon_release 17 (quincy) 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:election_strategy: 1 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:38:59.932 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-23T16:38:59.933 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T16:38:59.981 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0469449d-bc82-4171-9959-1bb87fc3dec3... 2024-04-23T16:38:59.982 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T16:39:00.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.052 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T16:39:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = "172.21.15.79,[v2:172.21.15.79:3301,v1:172.21.15.79:6790],172.21.15.204" 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: mon client directed command retry = 5 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: mon election default strategy = 1 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: ms bind msgr1 = False 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: ms bind msgr2 = True 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject delay max = 1 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject delay probability = 0.005 2024-04-23T16:39:00.055 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject delay type = mon 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject internal delays = 0.002 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject socket failures = 2500 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: ms type = async 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 600 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2024-04-23T16:39:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug trim objects = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.057 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression algorithm = zstd 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression mode = aggressive 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore zero block detection = True 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 20 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 20 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2024-04-23T16:39:00.058 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 10 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify cached snaps = True 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify missing on start = True 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock override recovery settings = True 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue = debug_random 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue cut off = debug_random 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-04-23T16:39:00.059 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 10 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2024-04-23T16:39:00.060 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2024-04-23T16:39:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon min osdmap epochs = 25 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune interval = 2 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune min = 15 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune txsize = 2 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon scrub inject crc mismatch = 0.01 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon scrub inject missing keys = 0.05 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: mon scrub interval = 300 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: paxos min = 10 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: paxos service trim min = 5 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: paxos trim min = 10 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:00.062 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.c] 2024-04-23T16:39:00.063 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.b] 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-04-23T16:39:00.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-04-23T16:39:00.065 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.066 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = "172.21.15.79,[v2:172.21.15.79:3301,v1:172.21.15.79:6790],172.21.15.204" 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: mon election default strategy = 1 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms bind msgr1 = False 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms bind msgr2 = True 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay max = 1 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay probability = 0.005 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay type = mon 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject internal delays = 0.002 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 2500 2024-04-23T16:39:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: ms type = async 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-04-23T16:39:00.068 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug trim objects = true 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-04-23T16:39:00.069 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore compression algorithm = zstd 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore compression mode = aggressive 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore zero block detection = True 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 20 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 20 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 10 2024-04-23T16:39:00.070 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify cached snaps = True 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify missing on start = True 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock override recovery settings = True 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue = debug_random 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue cut off = debug_random 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-04-23T16:39:00.071 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 10 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-04-23T16:39:00.072 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-04-23T16:39:00.073 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon min osdmap epochs = 25 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune interval = 2 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune min = 15 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune txsize = 2 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon scrub inject crc mismatch = 0.01 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon scrub inject missing keys = 0.05 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: mon scrub interval = 300 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: paxos min = 10 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: paxos service trim min = 5 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: paxos trim min = 10 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-04-23T16:39:00.074 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 20 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.c] 2024-04-23T16:39:00.075 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.b] 2024-04-23T16:39:00.076 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-23T16:39:00.077 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-23T16:39:00.170 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-23T16:39:00.170 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:00.170 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-23T16:39:00.220 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:00.221 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-23T16:39:00.268 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:39:00.268 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:00.268 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T16:39:00.268 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T16:39:00.332 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:00.332 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:00.381 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:39:00.381 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:00.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T16:39:00.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T16:39:00.404 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:00.404 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:00.451 INFO:tasks.ceph:Setting up mon nodes... 2024-04-23T16:39:00.452 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-23T16:39:00.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T16:39:00.537 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T16:39:00.545 INFO:tasks.ceph:Setting up mds nodes... 2024-04-23T16:39:00.545 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-23T16:39:00.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T16:39:00.620 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-23T16:39:00.632 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-23T16:39:00.632 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-23T16:39:00.632 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-23T16:39:00.646 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-23T16:39:00.646 INFO:tasks.ceph:role: osd.0 2024-04-23T16:39:00.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:39:00.646 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T16:39:00.708 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T16:39:00.709 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T16:39:00.709 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T16:39:01.197 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-23T16:39:01.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-23T16:39:01.200 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-23T16:39:01.362 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-23T16:39:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:39:01.374 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T16:39:01.374 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-23T16:39:01.428 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-23T16:39:01.428 INFO:tasks.ceph:role: osd.1 2024-04-23T16:39:01.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:39:01.429 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T16:39:01.492 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T16:39:01.492 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T16:39:01.492 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T16:39:01.493 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T16:39:01.940 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-23T16:39:01.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-23T16:39:01.942 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-23T16:39:02.039 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-23T16:39:02.094 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T16:39:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:39:02.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-23T16:39:02.152 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-23T16:39:02.152 INFO:tasks.ceph:role: osd.2 2024-04-23T16:39:02.152 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:39:02.152 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T16:39:02.216 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T16:39:02.216 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T16:39:02.216 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T16:39:02.217 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T16:39:02.670 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-23T16:39:02.672 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-23T16:39:02.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-23T16:39:02.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-23T16:39:02.796 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T16:39:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:39:02.797 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:02.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:02.902+0000 7ff66618a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-23T16:39:02.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:02.902+0000 7ff66618a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-23T16:39:02.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:02.902+0000 7ff66618a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T16:39:02.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:02.902+0000 7ff66618a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T16:39:02.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:02.902+0000 7ff66618a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T16:39:02.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:02.910+0000 7ff66618a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-23T16:39:05.901 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T16:39:05.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:05.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:05.994+0000 7f61a2ef47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-23T16:39:05.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:05.994+0000 7f61a2ef47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-23T16:39:05.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:05.994+0000 7f61a2ef47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T16:39:05.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:05.994+0000 7f61a2ef47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T16:39:05.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:05.994+0000 7f61a2ef47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T16:39:06.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:05.994+0000 7f61a2ef47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-23T16:39:08.984 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T16:39:09.041 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:09.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:09.118+0000 7ff16ad6d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-23T16:39:09.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:09.118+0000 7ff16ad6d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-23T16:39:09.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:09.118+0000 7ff16ad6d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T16:39:09.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:09.118+0000 7ff16ad6d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T16:39:09.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:09.118+0000 7ff16ad6d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T16:39:09.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:09.122+0000 7ff16ad6d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-23T16:39:12.099 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T16:39:12.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-23T16:39:12.167 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-23T16:39:12.167 INFO:tasks.ceph:role: osd.3 2024-04-23T16:39:12.167 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:39:12.167 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T16:39:12.230 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T16:39:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T16:39:12.711 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-04-23T16:39:12.712 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-04-23T16:39:12.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-23T16:39:12.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-23T16:39:12.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T16:39:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:39:12.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-23T16:39:12.911 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-23T16:39:12.911 INFO:tasks.ceph:role: osd.4 2024-04-23T16:39:12.911 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:39:12.911 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T16:39:12.975 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T16:39:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T16:39:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T16:39:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T16:39:12.975 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T16:39:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-04-23T16:39:12.976 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T16:39:12.976 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T16:39:12.976 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T16:39:12.976 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T16:39:13.401 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-04-23T16:39:13.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-04-23T16:39:13.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-23T16:39:13.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-23T16:39:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:39:13.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T16:39:13.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-23T16:39:13.572 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-23T16:39:13.572 INFO:tasks.ceph:role: osd.5 2024-04-23T16:39:13.572 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:39:13.572 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T16:39:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T16:39:13.636 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T16:39:14.105 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-04-23T16:39:14.107 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-04-23T16:39:14.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-23T16:39:14.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-23T16:39:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:39:14.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T16:39:14.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:14.327 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:14.324+0000 7f90c5d1a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-23T16:39:14.329 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:14.324+0000 7f90c5d1a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-23T16:39:14.329 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:14.324+0000 7f90c5d1a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T16:39:14.329 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:14.324+0000 7f90c5d1a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T16:39:14.329 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:14.324+0000 7f90c5d1a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T16:39:14.329 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:14.324+0000 7f90c5d1a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-23T16:39:17.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T16:39:17.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:17.409 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:17.404+0000 7fbcf07967c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-23T16:39:17.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:17.404+0000 7fbcf07967c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-23T16:39:17.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:17.404+0000 7fbcf07967c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T16:39:17.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:17.404+0000 7fbcf07967c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T16:39:17.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:17.404+0000 7fbcf07967c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T16:39:17.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:17.404+0000 7fbcf07967c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-23T16:39:20.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T16:39:20.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:20.539 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:20.536+0000 7fafec14a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-23T16:39:20.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:20.536+0000 7fafec14a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-23T16:39:20.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:20.536+0000 7fafec14a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T16:39:20.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:20.536+0000 7fafec14a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T16:39:20.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:20.536+0000 7fafec14a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T16:39:20.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:20.536+0000 7fafec14a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-23T16:39:23.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T16:39:23.571 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-23T16:39:23.571 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:23.571 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-23T16:39:23.585 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:23.585 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-23T16:39:23.639 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:23.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-23T16:39:23.691 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:23.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-23T16:39:23.704 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:23.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-23T16:39:23.759 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:23.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-23T16:39:23.814 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:23.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-23T16:39:23.871 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:23.871 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-23T16:39:23.919 INFO:tasks.ceph:Adding keys to all mons... 2024-04-23T16:39:23.919 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T16:39:23.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAm5CdmQ+cNNhAAyjHD9EstDfdMOWy8coyifA== 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAp5CdmHr2ZOxAAiU97i2WlqtJfm7fEHd9SoQ== 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAt5CdmF+VnBxAAE263ewGq4NAPXOBoTTxQWw== 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAk5CdmPocJIBAAlUe1xznJYdQdP6GiquOXLA== 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAy5Cdm0OCLExAA7//uh2W6sS+8D3mj7rvkVw== 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.4] 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQA15CdmbENuGBAAOExgbsXdIAXohZDD/oC1HA== 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.5] 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQA45Cdm38AoIBAA/P1fNT36Is/3oxuDDbPn+Q== 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-04-23T16:39:23.966 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAk5CdmTbL7JBAAtmPXzBJyQnplcsL2TUjFxQ== 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAm5CdmQ+cNNhAAyjHD9EstDfdMOWy8coyifA== 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAp5CdmHr2ZOxAAiU97i2WlqtJfm7fEHd9SoQ== 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAt5CdmF+VnBxAAE263ewGq4NAPXOBoTTxQWw== 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAk5CdmPocJIBAAlUe1xznJYdQdP6GiquOXLA== 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAy5Cdm0OCLExAA7//uh2W6sS+8D3mj7rvkVw== 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2024-04-23T16:39:23.974 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA15CdmbENuGBAAOExgbsXdIAXohZDD/oC1HA== 2024-04-23T16:39:23.975 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2024-04-23T16:39:23.975 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA45Cdm38AoIBAA/P1fNT36Is/3oxuDDbPn+Q== 2024-04-23T16:39:23.975 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-04-23T16:39:23.975 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAk5CdmTbL7JBAAtmPXzBJyQnplcsL2TUjFxQ== 2024-04-23T16:39:23.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.062 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.147 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.234 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T16:39:24.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T16:39:24.318 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.402 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.481 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T16:39:24.561 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T16:39:24.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T16:39:24.650 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-23T16:39:24.650 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-23T16:39:24.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T16:39:24.749 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-23T16:39:24.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-23T16:39:24.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T16:39:24.903 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-23T16:39:24.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-23T16:39:24.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T16:39:25.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-23T16:39:25.037 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:25.040 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T16:39:25.082 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-23T16:39:25.082 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-23T16:39:25.083 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-23T16:39:25.089 INFO:tasks.ceph.mon.a:Started 2024-04-23T16:39:25.089 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-23T16:39:25.090 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-23T16:39:25.092 INFO:tasks.ceph.mon.c:Started 2024-04-23T16:39:25.092 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-23T16:39:25.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-23T16:39:25.124 INFO:tasks.ceph.mon.b:Started 2024-04-23T16:39:25.124 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-23T16:39:25.124 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-23T16:39:25.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-23T16:39:25.126 INFO:tasks.ceph.mgr.x:Started 2024-04-23T16:39:25.126 INFO:tasks.ceph:Setting crush tunables to default 2024-04-23T16:39:25.127 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd crush tunables default 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient: get_monmap_and_config 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient: build_initial_monmap 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient: monmap: 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:25.243744+0000 2024-04-23T16:39:25.242 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:25.243744+0000 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:25.243 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:25.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 1 Processor -- start 2024-04-23T16:39:25.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 1 -- start start 2024-04-23T16:39:25.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient: init 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f406c0a3250 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient(hunting): picked mon.noname-a con 0x7f406c0a3630 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f406c0a3ac0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c0a14f0 con 0x7f406c0a3250 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c1382b0 con 0x7f406c0a3ac0 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0a1760 con 0x7f406c0a3630 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f407358d640 10 monclient(hunting): authenticate will time out at 919.431331s 2024-04-23T16:39:25.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4071329640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f406c0a3250 0x7f406c137ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:47504/0 (socket says 172.21.15.79:47504) 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 learned_addr learned my addr 172.21.15.79:0/46835975 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f406c0a3630 auth_method 0 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.238+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.242+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v1:172.21.15.79:6789/0 conn(0x7f406c0a3250 legacy=0x7f406c137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:25.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.242+0000 7f4071b2a640 1 -- 172.21.15.79:0/46835975 >> v1:172.21.15.204:6789/0 conn(0x7f406c0a3ac0 legacy=0x7f406c13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:25.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f406c0a3630 auth_method 0 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:25.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.442+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f406c0a3630 auth_method 0 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:25.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:25.846+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T16:39:26.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f406c0a3630 auth_method 0 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:26.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:26.650+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T16:39:28.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): tick 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 monclient(hunting): continuing hunt 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 msgr2=0x7f406c13c8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.79:3300/0 conn(0x7f406c0a3630 0x7f406c13c8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 >> v1:172.21.15.204:6789/0 conn(0x7f406c0a3ac0 legacy=0x7f406c13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 >> v1:172.21.15.79:6789/0 conn(0x7f406c0a3250 legacy=0x7f406c137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient: _add_conns ranks=[4,2] 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f40500037f0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:28.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f4050006330 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0a1760 con 0x7f40500037f0 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4050008f20 con 0x7f4050006330 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4062ffd640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f4050006330 auth_method 0 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 10 monclient(hunting): get_auth_request con 0x7f40500037f0 auth_method 0 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:28.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:28.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.242+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f4050006330 auth_method 0 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 10 monclient(hunting): get_auth_request con 0x7f40500037f0 auth_method 0 2024-04-23T16:39:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.446+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T16:39:28.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:28.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f4050006330 auth_method 0 2024-04-23T16:39:28.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:28.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:28.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:28.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 10 monclient(hunting): get_auth_request con 0x7f40500037f0 auth_method 0 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:28.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:28.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T16:39:28.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:39:28.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:28.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:28.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:28.850+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f4050006330 auth_method 0 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 10 monclient(hunting): get_auth_request con 0x7f40500037f0 auth_method 0 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:29.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:29.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:29.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4070b28640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T16:39:29.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-23T16:39:29.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:39:29.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:29.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:29.654+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T16:39:30.819 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:30.812+0000 7f2dc5c80180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): tick 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 monclient(hunting): continuing hunt 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f40500087f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f40500087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 msgr2=0x7f4050005c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f40500037f0 0x7f4050005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f4050008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f4050006330 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071b2a640 1 -- 172.21.15.79:0/46835975 reap_dead start 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f406c0a3ac0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f4050003c40 0x7f405000caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f4050003c40 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c0a1a40 con 0x7f406c0a3ac0 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0a1760 con 0x7f4050003c40 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 1 -- 172.21.15.79:0/46835975 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4050008f20 con 0x7f4050006330 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4062ffd640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f4050008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): get_auth_request con 0x7f4050006330 auth_method 0 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071b2a640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f4050003c40 0x7f405000caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071b2a640 10 monclient(hunting): get_auth_request con 0x7f4050003c40 auth_method 0 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071b2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071b2a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071b2a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient(hunting): handle_auth_done global_id 4103 payload 274 2024-04-23T16:39:31.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f4050003c40 msgr2=0x7f405000caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f4050003c40 0x7f405000caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> v1:172.21.15.204:6789/0 conn(0x7f406c0a3ac0 legacy=0x7f405000b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient: _finish_hunting 0 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 1 monclient: found mon.noname-b 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:31.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.242+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405000e070 con 0x7f4050006330 2024-04-23T16:39:31.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4071329640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f4050008770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:31.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4071329640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f4050008770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4068007fd0 tx=0x7f406801d180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b1182e7090dd241 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 1 -- 172.21.15.79:0/46835975 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40680ab070 con 0x7f4050006330 2024-04-23T16:39:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: dump: 2024-04-23T16:39:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 1 -- 172.21.15.79:0/46835975 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40680a5de0 con 0x7f4050006330 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 1 -- 172.21.15.79:0/46835975 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40680af640 con 0x7f4050006330 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f4063fff640 10 monclient: dump: 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f407358d640 5 monclient: authenticate success, global_id 4103 2024-04-23T16:39:31.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f407358d640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:31.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.250+0000 7f407358d640 4 set_mon_vals no callback set 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient: shutdown 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 msgr2=0x7f4050008770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f4050008770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4068007fd0 tx=0x7f406801d180 comp rx=0 tx=0).stop 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- 172.21.15.79:0/46835975 shutdown_connections 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 --2- 172.21.15.79:0/46835975 >> v2:172.21.15.204:3300/0 conn(0x7f4050003c40 0x7f405000caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 --2- 172.21.15.79:0/46835975 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4050006330 0x7f4050008770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- 172.21.15.79:0/46835975 >> 172.21.15.79:0/46835975 conn(0x7f406c0aa290 msgr2=0x7f406c0a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- 172.21.15.79:0/46835975 shutdown_connections 2024-04-23T16:39:31.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- 172.21.15.79:0/46835975 wait complete. 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient: build_initial_monmap 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient: monmap: 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 Processor -- start 2024-04-23T16:39:31.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- start start 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient: init 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 0x7f406c1286f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): picked mon.noname-a con 0x7f4050003c40 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4050006330 0x7f406c1d0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): picked mon.noname-b con 0x7f4050006330 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f406c0a3ac0 0x7f406c123db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): picked mon.noname-c con 0x7f406c0a3ac0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0a5240 con 0x7f4050003c40 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0a50c0 con 0x7f4050006330 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f406c0a53c0 con 0x7f406c0a3ac0 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f407358d640 10 monclient(hunting): authenticate will time out at 925.447683s 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 0x7f406c1286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 0x7f406c1286f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45362/0 (socket says 172.21.15.79:45362) 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 1 -- 172.21.15.79:0/3154616716 learned_addr learned my addr 172.21.15.79:0/3154616716 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 10 monclient(hunting): get_auth_request con 0x7f4050003c40 auth_method 0 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071b2a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f406c0a3ac0 0x7f406c123db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071329640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.254+0000 7f4071b2a640 10 monclient(hunting): get_auth_request con 0x7f406c0a3ac0 auth_method 0 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:31.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4050006330 0x7f406c1d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): get_auth_request con 0x7f4050006330 auth_method 0 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4070b28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 -- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f406c0a3ac0 msgr2=0x7f406c123db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f406c0a3ac0 0x7f406c123db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 -- 172.21.15.79:0/3154616716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4050006330 msgr2=0x7f406c1d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4050006330 0x7f406c1d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 10 monclient: _finish_hunting 0 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 monclient: found mon.noname-a 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 -- 172.21.15.79:0/3154616716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c125e40 con 0x7f4050003c40 2024-04-23T16:39:31.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071b2a640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f406c0a3ac0 0x7f406c123db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 -- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 msgr2=0x7f406c1286f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4071329640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 0x7f406c1286f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f40680010f0 tx=0x7f406801b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40680bd020 con 0x7f4050003c40 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: dump: 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.268 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4068005000 con 0x7f4050003c40 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 5 monclient: authenticate success, global_id 4104 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 10 monclient: _renew_subs 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c0a3ff0 con 0x7f4050003c40 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40680ab040 con 0x7f4050003c40 2024-04-23T16:39:31.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 10 monclient: dump: 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f40637fe640 4 set_mon_vals no callback set 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 10 monclient: _renew_subs 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:31.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c127a10 con 0x7f4050003c40 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f40680c0020 con 0x7f4050003c40 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.258+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f40680b9850 con 0x7f4050003c40 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f407358d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f407358d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f407358d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c0692e0 con 0x7f4050003c40 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f40680c7030 con 0x7f4050003c40 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f4061ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:31.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.262+0000 7f4061ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:31.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:31.300+0000 7f2dc5c80180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T16:39:31.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.326+0000 7f407358d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-23T16:39:31.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.326+0000 7f407358d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-23T16:39:31.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.326+0000 7f407358d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:31.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.326+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f406c09bfb0 con 0x7f4050003c40 2024-04-23T16:39:31.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.414+0000 7f4061ffb640 1 -- 172.21.15.79:0/3154616716 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f4068016f10 con 0x7f4050003c40 2024-04-23T16:39:31.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.414+0000 7f4061ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-23T16:39:31.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.414+0000 7f4061ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-23T16:39:31.420 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 10 monclient: shutdown 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 msgr2=0x7f406c1286f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 0x7f406c1286f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f40680010f0 tx=0x7f406801b9c0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 shutdown_connections 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f406c0a3ac0 0x7f406c123db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4050006330 0x7f406c1d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 --2- 172.21.15.79:0/3154616716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4050003c40 0x7f406c1286f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 >> 172.21.15.79:0/3154616716 conn(0x7f406c0aa290 msgr2=0x7f406c0a7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 shutdown_connections 2024-04-23T16:39:31.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:31.418+0000 7f407358d640 1 -- 172.21.15.79:0/3154616716 wait complete. 2024-04-23T16:39:31.444 INFO:tasks.ceph:check_enable_crimson: False 2024-04-23T16:39:31.444 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-23T16:39:31.445 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:31.445 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-23T16:39:31.457 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:31.457 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-23T16:39:31.510 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:31.504+0000 7f2dc5c80180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T16:39:31.511 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-23T16:39:31.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-23T16:39:31.563 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:31.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-23T16:39:31.575 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:31.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-23T16:39:31.630 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T16:39:31.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-23T16:39:31.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 2728fa88-43d5-4a55-998c-4b58cfb56136 0 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient: get_monmap_and_config 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient: build_initial_monmap 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient: monmap: 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:39:31.840327+0000 2024-04-23T16:39:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:39:31.840327+0000 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:31.841 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 1 Processor -- start 2024-04-23T16:39:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 1 -- start start 2024-04-23T16:39:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient: init 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efca40a3600 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efca40a39e0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 0x7efca4140440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient(hunting): picked mon.noname-b con 0x7efca413e050 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca41383a0 con 0x7efca40a39e0 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca40a5d40 con 0x7efca40a3600 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efca40a0f10 con 0x7efca413e050 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efcaa8bd640 10 monclient(hunting): authenticate will time out at 915.950617s 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca3fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7efca40a3600 0x7efca4137c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:58662/0 (socket says 172.21.15.204:58662) 2024-04-23T16:39:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca3fff640 1 -- 172.21.15.204:0/584530755 learned_addr learned my addr 172.21.15.204:0/584530755 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 1 --2- 172.21.15.204:0/584530755 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 0x7efca4140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): get_auth_request con 0x7efca413e050 auth_method 0 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 1 -- 172.21.15.204:0/584530755 >> v1:172.21.15.204:6789/0 conn(0x7efca40a3600 legacy=0x7efca4137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 1 -- 172.21.15.204:0/584530755 >> v1:172.21.15.79:6789/0 conn(0x7efca40a39e0 legacy=0x7efca413c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient: _finish_hunting 0 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 1 monclient: found mon.noname-b 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.836+0000 7efca8e5a640 1 -- 172.21.15.204:0/584530755 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca40a1600 con 0x7efca413e050 2024-04-23T16:39:31.847 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 -- 172.21.15.204:0/584530755 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 msgr2=0x7efca4140440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:31.847 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 --2- 172.21.15.204:0/584530755 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 0x7efca4140440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009dc0 tx=0x7efc9c01c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbefad8e991dc74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 1 -- 172.21.15.204:0/584530755 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0b2070 con 0x7efca413e050 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: dump: 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:31.849 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 1 -- 172.21.15.204:0/584530755 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9c0a5840 con 0x7efca413e050 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 1 -- 172.21.15.204:0/584530755 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0a5c10 con 0x7efca413e050 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca2ffd640 10 monclient: dump: 2024-04-23T16:39:31.850 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 5 monclient: authenticate success, global_id 4109 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 4 set_mon_vals no callback set 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: shutdown 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:31.851 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- 172.21.15.204:0/584530755 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 msgr2=0x7efca4140440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 --2- 172.21.15.204:0/584530755 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 0x7efca4140440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009dc0 tx=0x7efc9c01c7d0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- 172.21.15.204:0/584530755 shutdown_connections 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 --2- 172.21.15.204:0/584530755 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca413e050 0x7efca4140440 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- 172.21.15.204:0/584530755 >> 172.21.15.204:0/584530755 conn(0x7efca40aa3e0 msgr2=0x7efca40a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- 172.21.15.204:0/584530755 shutdown_connections 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- 172.21.15.204:0/584530755 wait complete. 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: build_initial_monmap 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: monmap: 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:31.853 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:31.853 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:31.853 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:31.853 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 Processor -- start 2024-04-23T16:39:31.853 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- start start 2024-04-23T16:39:31.853 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: init 2024-04-23T16:39:31.856 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efca40a3600 0x7efca41cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): picked mon.noname-a con 0x7efca40a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca40a39e0 0x7efca41d0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): picked mon.noname-c con 0x7efca40a39e0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 0x7efca409cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): picked mon.noname-b con 0x7efca413e050 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efca4143730 con 0x7efca40a3600 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efca41435b0 con 0x7efca413e050 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efca41438b0 con 0x7efca40a39e0 2024-04-23T16:39:31.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efcaa8bd640 10 monclient(hunting): authenticate will time out at 915.958541s 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 0x7efca409cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 0x7efca409cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56592/0 (socket says 172.21.15.204:56592) 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 -- 172.21.15.204:0/3762973585 learned_addr learned my addr 172.21.15.204:0/3762973585 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): get_auth_request con 0x7efca413e050 auth_method 0 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca3fff640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efca40a3600 0x7efca41cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-04-23T16:39:31.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 -- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca40a39e0 msgr2=0x7efca41d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca37fe640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca40a39e0 0x7efca41d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca40a39e0 0x7efca41d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 -- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efca40a3600 msgr2=0x7efca41cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efca40a3600 0x7efca41cfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient: _finish_hunting 0 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 monclient: found mon.noname-b 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca8e5a640 1 -- 172.21.15.204:0/3762973585 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca40a8370 con 0x7efca413e050 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca37fe640 10 monclient: get_auth_request con 0x7efca40a39e0 auth_method 0 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca3fff640 10 monclient: get_auth_request con 0x7efca40a3600 auth_method 0 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca37fe640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca40a39e0 0x7efca41d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.844+0000 7efca3fff640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efca40a3600 0x7efca41cfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca8e5a640 1 -- 172.21.15.204:0/3762973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 msgr2=0x7efca409cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:31.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca8e5a640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 0x7efca409cbd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0b0040 tx=0x7efc9c019570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0bd020 con 0x7efca413e050 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: dump: 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: _finish_auth 0 2024-04-23T16:39:31.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9c0b8880 con 0x7efca413e050 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0b2040 con 0x7efca413e050 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 10 monclient: dump: 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:31.861 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca27fc640 4 set_mon_vals no callback set 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 5 monclient: authenticate success, global_id 4108 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 monclient: _renew_subs 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca409e870 con 0x7efca413e050 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 monclient: _renew_subs 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca4137020 con 0x7efca413e050 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efc9c0c0020 con 0x7efca413e050 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7efc9c0044b0 con 0x7efca413e050 2024-04-23T16:39:31.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:31.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:31.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:31.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.848+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca4068ae0 con 0x7efca413e050 2024-04-23T16:39:31.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.860+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efc9c0c7070 con 0x7efca413e050 2024-04-23T16:39:31.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.860+0000 7efca0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:31.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.860+0000 7efca0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:31.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.928+0000 7efcaa8bd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2728fa88-43d5-4a55-998c-4b58cfb56136", "id": 0}] 2024-04-23T16:39:31.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.928+0000 7efcaa8bd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2728fa88-43d5-4a55-998c-4b58cfb56136", "id": 0}] 2024-04-23T16:39:31.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.928+0000 7efcaa8bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:31.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:31.928+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2728fa88-43d5-4a55-998c-4b58cfb56136", "id": 0} v 0) -- 0x7efca40a7ce0 con 0x7efca413e050 2024-04-23T16:39:32.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.072+0000 7f2dc5c80180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T16:39:32.122 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.116+0000 7efca0ff9640 1 -- 172.21.15.204:0/3762973585 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2728fa88-43d5-4a55-998c-4b58cfb56136", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7efc9c0164e0 con 0x7efca413e050 2024-04-23T16:39:32.122 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.116+0000 7efca0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2728fa88-43d5-4a55-998c-4b58cfb56136", "id": 0}] 2024-04-23T16:39:32.122 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.116+0000 7efca0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:32.122 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-04-23T16:39:32.124 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 10 monclient: shutdown 2024-04-23T16:39:32.124 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.124 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 msgr2=0x7efca409cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.124 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 0x7efca409cbd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0b0040 tx=0x7efc9c019570 comp rx=0 tx=0).stop 2024-04-23T16:39:32.124 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 shutdown_connections 2024-04-23T16:39:32.124 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efca413e050 0x7efca409cbd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.125 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efca40a39e0 0x7efca41d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.125 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 --2- 172.21.15.204:0/3762973585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efca40a3600 0x7efca41cfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.125 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 >> 172.21.15.204:0/3762973585 conn(0x7efca40aa3e0 msgr2=0x7efca40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.125 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 shutdown_connections 2024-04-23T16:39:32.125 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.120+0000 7efcaa8bd640 1 -- 172.21.15.204:0/3762973585 wait complete. 2024-04-23T16:39:32.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.124+0000 7f2dc5c80180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T16:39:32.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 5dcf8600-d4ef-47a3-8e79-a9fde5ab876c 1 2024-04-23T16:39:32.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.180+0000 7f2dc5c80180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T16:39:32.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient: get_monmap_and_config 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient: monmap: 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:39:32.253066+0000 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:39:32.253066+0000 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:32.253 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 1 Processor -- start 2024-04-23T16:39:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 1 -- start start 2024-04-23T16:39:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient: init 2024-04-23T16:39:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f898c135910 0x7f898c0a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient(hunting): picked mon.noname-a con 0x7f898c135910 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f898c135cf0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f898c136130 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f898c09ffd0 con 0x7f898c135cf0 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f898c1354e0 con 0x7f898c136130 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f898c0a1b00 con 0x7f898c135910 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f89945ef640 10 monclient(hunting): authenticate will time out at 916.363112s 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f8992b8c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f898c136130 0x7f898c09f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:58670/0 (socket says 172.21.15.204:58670) 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 learned_addr learned my addr 172.21.15.204:0/3360657599 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 1 --2- 172.21.15.204:0/3360657599 >> v2:172.21.15.79:3300/0 conn(0x7f898c135910 0x7f898c0a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): get_auth_request con 0x7f898c135910 auth_method 0 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.248+0000 7f899238b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.260 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 >> v1:172.21.15.204:6789/0 conn(0x7f898c136130 legacy=0x7f898c09f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.260 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579032408 0 0) 0x7f898c1354e0 con 0x7f898c136130 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): _negotiate 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): my global_id is 4111 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8978003530 con 0x7f898c136130 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2033114232 0 0) 0x7f8978003530 con 0x7f898c136130 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 >> v2:172.21.15.79:3300/0 conn(0x7f898c135910 msgr2=0x7f898c0a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 --2- 172.21.15.204:0/3360657599 >> v2:172.21.15.79:3300/0 conn(0x7f898c135910 0x7f898c0a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 >> v1:172.21.15.79:6789/0 conn(0x7f898c135cf0 legacy=0x7f898c09f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 monclient: found mon.noname-c-legacy 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.204:6789/0 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c0a1820 con 0x7f898c136130 2024-04-23T16:39:32.261 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1099920027 0 0) 0x7f8984001d70 con 0x7f898c136130 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f89945ef640 5 monclient: authenticate success, global_id 4111 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f89945ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: dump: 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.262 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] but i'm connected to v1:172.21.15.204:6789/0, reconnecting 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 >> v1:172.21.15.204:6789/0 conn(0x7f898c136130 legacy=0x7f898c09f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89780039c0 0x7f8978003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): picked mon.a con 0x7f89780039c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978004810 0x7f8978004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): picked mon.c con 0x7f8978004810 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.263 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 0x7f897800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): picked mon.b con 0x7f8978005240 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f897800c170 con 0x7f89780039c0 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f897800c320 con 0x7f8978005240 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f897800c520 con 0x7f8978004810 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991389640 10 monclient(hunting): _finish_auth 0 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f89945ef640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 0x7f897800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): get_auth_request con 0x7f8978005240 auth_method 0 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f899238b640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89780039c0 0x7f8978003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient(hunting): handle_auth_done global_id 4111 payload 293 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978004810 msgr2=0x7f8978004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8991b8a640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978004810 0x7f8978004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978004810 0x7f8978004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89780039c0 msgr2=0x7f8978003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89780039c0 0x7f8978003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 monclient: found mon.b 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8978003530 con 0x7f8978005240 2024-04-23T16:39:32.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f8992b8c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f899238b640 10 monclient: get_auth_request con 0x7f89780039c0 auth_method 0 2024-04-23T16:39:32.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.252+0000 7f899238b640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89780039c0 0x7f8978003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:32.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 reap_dead start 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8992b8c640 1 -- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 msgr2=0x7f897800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 0x7f897800b9d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f898400ab00 tx=0x7f898401d560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6212d34dfbc9c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898400a830 con 0x7f8978005240 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: dump: 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.271 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89840aa3f0 con 0x7f8978005240 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 1 -- 172.21.15.204:0/3360657599 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840aa7e0 con 0x7f8978005240 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 4 set_mon_vals no callback set 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f8991389640 10 monclient: dump: 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.272 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 10 monclient: shutdown 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 -- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 msgr2=0x7f897800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 0x7f897800b9d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f898400ab00 tx=0x7f898401d560 comp rx=0 tx=0).stop 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 -- 172.21.15.204:0/3360657599 shutdown_connections 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 --2- 172.21.15.204:0/3360657599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8978005240 0x7f897800b9d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 -- 172.21.15.204:0/3360657599 >> 172.21.15.204:0/3360657599 conn(0x7f898c0a8680 msgr2=0x7f898c0a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 -- 172.21.15.204:0/3360657599 shutdown_connections 2024-04-23T16:39:32.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.256+0000 7f89945ef640 1 -- 172.21.15.204:0/3360657599 wait complete. 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient: monmap: 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 Processor -- start 2024-04-23T16:39:32.274 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 -- start start 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient: init 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978005240 0x7f898c11e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): picked mon.noname-c con 0x7f8978005240 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f898c1cf960 0x7f898c13b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): picked mon.noname-a con 0x7f898c1cf960 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 0x7f898c13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): picked mon.noname-b con 0x7f898c1cfda0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f898c0a0ad0 con 0x7f898c1cf960 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f898c0a0950 con 0x7f898c1cfda0 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.275 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f898c0a0c50 con 0x7f8978005240 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient(hunting): authenticate will time out at 916.372969s 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 0x7f898c13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 0x7f898c13ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56614/0 (socket says 172.21.15.204:56614) 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 -- 172.21.15.204:0/3463388214 learned_addr learned my addr 172.21.15.204:0/3463388214 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): get_auth_request con 0x7f898c1cfda0 auth_method 0 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f899238b640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978005240 0x7f898c11e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8991b8a640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f898c1cf960 0x7f898c13b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f899238b640 10 monclient(hunting): get_auth_request con 0x7f8978005240 auth_method 0 2024-04-23T16:39:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f899238b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f899238b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f899238b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8991b8a640 10 monclient(hunting): get_auth_request con 0x7f898c1cf960 auth_method 0 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8991b8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8991b8a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8991b8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 -- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978005240 msgr2=0x7f898c11e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978005240 0x7f898c11e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 -- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f898c1cf960 msgr2=0x7f898c13b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f898c1cf960 0x7f898c13b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 monclient: found mon.noname-b 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 -- 172.21.15.204:0/3463388214 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c11f8d0 con 0x7f898c1cfda0 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 -- 172.21.15.204:0/3463388214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 msgr2=0x7f898c13ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8992b8c640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 0x7f898c13ba30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f898401de00 tx=0x7f8978005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840016d0 con 0x7f898c1cfda0 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: dump: 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.278 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89840bd070 con 0x7f898c1cfda0 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840aa7e0 con 0x7f898c1cfda0 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f897effd640 10 monclient: dump: 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.279 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f8990b88640 4 set_mon_vals no callback set 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 5 monclient: authenticate success, global_id 4114 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.260+0000 7f89945ef640 10 monclient: _renew_subs 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898c13d5e0 con 0x7f898c1cfda0 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 10 monclient: _renew_subs 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898c11f5f0 con 0x7f898c1cfda0 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f89840aa980 con 0x7f898c1cfda0 2024-04-23T16:39:32.280 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f89840b7d70 con 0x7f898c1cfda0 2024-04-23T16:39:32.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f2dc5c80180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f898c068ae0 con 0x7f898c1cfda0 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f898400d040 con 0x7f898c1cfda0 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f897effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.264+0000 7f897effd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:32.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.324+0000 7f2dc5c80180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T16:39:32.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.344+0000 7f89945ef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5dcf8600-d4ef-47a3-8e79-a9fde5ab876c", "id": 1}] 2024-04-23T16:39:32.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.344+0000 7f89945ef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5dcf8600-d4ef-47a3-8e79-a9fde5ab876c", "id": 1}] 2024-04-23T16:39:32.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.344+0000 7f89945ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.349 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.344+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5dcf8600-d4ef-47a3-8e79-a9fde5ab876c", "id": 1} v 0) -- 0x7f898c13d140 con 0x7f898c1cfda0 2024-04-23T16:39:32.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f897effd640 1 -- 172.21.15.204:0/3463388214 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5dcf8600-d4ef-47a3-8e79-a9fde5ab876c", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f89840b7470 con 0x7f898c1cfda0 2024-04-23T16:39:32.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f897effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5dcf8600-d4ef-47a3-8e79-a9fde5ab876c", "id": 1}] 2024-04-23T16:39:32.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f897effd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:32.356 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 10 monclient: shutdown 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 msgr2=0x7f898c13ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 0x7f898c13ba30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f898401de00 tx=0x7f8978005840 comp rx=0 tx=0).stop 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 shutdown_connections 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f898c1cfda0 0x7f898c13ba30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f898c1cf960 0x7f898c13b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 --2- 172.21.15.204:0/3463388214 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8978005240 0x7f898c11e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 >> 172.21.15.204:0/3463388214 conn(0x7f898c0a8680 msgr2=0x7f898c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 shutdown_connections 2024-04-23T16:39:32.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.352+0000 7f89945ef640 1 -- 172.21.15.204:0/3463388214 wait complete. 2024-04-23T16:39:32.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new d079e76b-d697-4efc-a10e-52178c04b550 2 2024-04-23T16:39:32.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.408+0000 7f2dc5c80180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T16:39:32.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.464+0000 7f2dc5c80180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T16:39:32.484 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient: get_monmap_and_config 2024-04-23T16:39:32.484 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.484 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient: monmap: 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:39:32.484947+0000 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:39:32.484947+0000 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:32.485 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 1 Processor -- start 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 1 -- start start 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient: init 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efeb40a25b0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efeb4075a40 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:32.487 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7efeb4075eb0 0x7efeb41401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient(hunting): picked mon.noname-a con 0x7efeb4075eb0 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb40a2ce0 con 0x7efeb40a25b0 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb40a0c00 con 0x7efeb4075a40 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7efeb40a07c0 con 0x7efeb4075eb0 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb90ee640 10 monclient(hunting): authenticate will time out at 916.594972s 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb259c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7efeb4075a40 0x7efeb413caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:58678/0 (socket says 172.21.15.204:58678) 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb259c640 1 -- 172.21.15.204:0/3069295194 learned_addr learned my addr 172.21.15.204:0/3069295194 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 1 --2- 172.21.15.204:0/3069295194 >> v2:172.21.15.79:3300/0 conn(0x7efeb4075eb0 0x7efeb41401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): get_auth_request con 0x7efeb4075eb0 auth_method 0 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.488 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.489 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.489 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.489 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.480+0000 7efeb359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb259c640 1 -- 172.21.15.204:0/3069295194 >> v1:172.21.15.204:6789/0 conn(0x7efeb4075a40 legacy=0x7efeb413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224437139 0 0) 0x7efeb40a0c00 con 0x7efeb4075a40 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): _negotiate 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): my global_id is 4117 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe9c003530 con 0x7efeb4075a40 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1832718457 0 0) 0x7efe9c003530 con 0x7efeb4075a40 2024-04-23T16:39:32.492 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb2d9d640 1 -- 172.21.15.204:0/3069295194 >> v1:172.21.15.79:6789/0 conn(0x7efeb40a25b0 legacy=0x7efeb4075330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 >> v2:172.21.15.79:3300/0 conn(0x7efeb4075eb0 msgr2=0x7efeb41401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 --2- 172.21.15.204:0/3069295194 >> v2:172.21.15.79:3300/0 conn(0x7efeb4075eb0 0x7efeb41401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 >> v1:172.21.15.79:6789/0 conn(0x7efeb40a25b0 legacy=0x7efeb4075330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 monclient: found mon.noname-c-legacy 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.204:6789/0 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb40a0ee0 con 0x7efeb4075a40 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1099920027 0 0) 0x7efea8001930 con 0x7efeb4075a40 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.493 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: dump: 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] but i'm connected to v1:172.21.15.204:6789/0, reconnecting 2024-04-23T16:39:32.494 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 >> v1:172.21.15.204:6789/0 conn(0x7efeb4075a40 legacy=0x7efeb413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efe9c0039c0 0x7efe9c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): picked mon.c con 0x7efe9c0039c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe9c004810 0x7efe9c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): picked mon.a con 0x7efe9c004810 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efe9c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): picked mon.b con 0x7efe9c0052f0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c00c190 con 0x7efe9c004810 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c00c340 con 0x7efe9c0052f0 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efe9c00c540 con 0x7efe9c0039c0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb1d9b640 10 monclient(hunting): _finish_auth 0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efe9c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): get_auth_request con 0x7efe9c0052f0 auth_method 0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb2d9d640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efe9c0039c0 0x7efe9c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb259c640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe9c004810 0x7efe9c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb2d9d640 10 monclient(hunting): get_auth_request con 0x7efe9c0039c0 auth_method 0 2024-04-23T16:39:32.496 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient(hunting): handle_auth_done global_id 4117 payload 293 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 -- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efe9c0039c0 msgr2=0x7efe9c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efe9c0039c0 0x7efe9c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 -- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe9c004810 msgr2=0x7efe9c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe9c004810 0x7efe9c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 monclient: found mon.b 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 1 -- 172.21.15.204:0/3069295194 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9c003530 con 0x7efe9c0052f0 2024-04-23T16:39:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb359e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb90ee640 5 monclient: authenticate success, global_id 4117 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb90ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb259c640 10 monclient: get_auth_request con 0x7efe9c004810 auth_method 0 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.484+0000 7efeb259c640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe9c004810 0x7efe9c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb359e640 1 -- 172.21.15.204:0/3069295194 reap_dead start 2024-04-23T16:39:32.502 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb359e640 1 -- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 msgr2=0x7efe9c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb359e640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efe9c00b9d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efeac009f90 tx=0x7efeac01cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5905e893c4c0cb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac009c10 con 0x7efe9c0052f0 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: dump: 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.503 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeac0a5c70 con 0x7efe9c0052f0 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb90ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb90ee640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb90ee640 4 set_mon_vals no callback set 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 1 -- 172.21.15.204:0/3069295194 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac003be0 con 0x7efe9c0052f0 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.488+0000 7efeb1d9b640 10 monclient: dump: 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.504 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient: shutdown 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 msgr2=0x7efe9c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efe9c00b9d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efeac009f90 tx=0x7efeac01cc60 comp rx=0 tx=0).stop 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- 172.21.15.204:0/3069295194 shutdown_connections 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 --2- 172.21.15.204:0/3069295194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efe9c00b9d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- 172.21.15.204:0/3069295194 >> 172.21.15.204:0/3069295194 conn(0x7efeb40a36a0 msgr2=0x7efeb40a3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.505 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- 172.21.15.204:0/3069295194 shutdown_connections 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- 172.21.15.204:0/3069295194 wait complete. 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient: monmap: 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:32.506 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 Processor -- start 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- start start 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient: init 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efeb413f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c0052f0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeb413cf10 0x7efeb41348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): picked mon.noname-a con 0x7efeb413cf10 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efeb41cf0c0 0x7efeb41cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): picked mon.noname-c con 0x7efeb41cf0c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40756c0 con 0x7efeb413cf10 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.507 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4075540 con 0x7efe9c0052f0 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efeb4075840 con 0x7efeb41cf0c0 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb90ee640 10 monclient(hunting): authenticate will time out at 916.606035s 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efeb413f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efeb413f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56644/0 (socket says 172.21.15.204:56644) 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 -- 172.21.15.204:0/4094764167 learned_addr learned my addr 172.21.15.204:0/4094764167 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): get_auth_request con 0x7efe9c0052f0 auth_method 0 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.508 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 -- 172.21.15.204:0/4094764167 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efeb41cf0c0 msgr2=0x7efeb41cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efeb41cf0c0 0x7efeb41cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 -- 172.21.15.204:0/4094764167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeb413cf10 msgr2=0x7efeb41348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeb413cf10 0x7efeb41348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 monclient: found mon.noname-b 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.492+0000 7efeb2d9d640 1 -- 172.21.15.204:0/4094764167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb411df80 con 0x7efe9c0052f0 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb2d9d640 1 -- 172.21.15.204:0/4094764167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 msgr2=0x7efeb413f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb2d9d640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efeb413f120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efea4007fd0 tx=0x7efea400e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4019070 con 0x7efe9c0052f0 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.509 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: dump: 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.510 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea40074f0 con 0x7efe9c0052f0 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 5 monclient: authenticate success, global_id 4120 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea40059f0 con 0x7efe9c0052f0 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 monclient: _renew_subs 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb4136ca0 con 0x7efe9c0052f0 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb159a640 4 set_mon_vals no callback set 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 10 monclient: dump: 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.511 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 monclient: _renew_subs 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb4072e70 con 0x7efe9c0052f0 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efea401c070 con 0x7efe9c0052f0 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7efea401e440 con 0x7efe9c0052f0 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.513 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.513 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.496+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb411db40 con 0x7efe9c0052f0 2024-04-23T16:39:32.513 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.500+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efea4005b90 con 0x7efe9c0052f0 2024-04-23T16:39:32.513 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.500+0000 7efe9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.513 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.500+0000 7efe9b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:32.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.568+0000 7efeb90ee640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d079e76b-d697-4efc-a10e-52178c04b550", "id": 2}] 2024-04-23T16:39:32.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.568+0000 7efeb90ee640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d079e76b-d697-4efc-a10e-52178c04b550", "id": 2}] 2024-04-23T16:39:32.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.568+0000 7efeb90ee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.568+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d079e76b-d697-4efc-a10e-52178c04b550", "id": 2} v 0) -- 0x7efeb413f6c0 con 0x7efe9c0052f0 2024-04-23T16:39:32.577 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efe9b7fe640 1 -- 172.21.15.204:0/4094764167 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d079e76b-d697-4efc-a10e-52178c04b550", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7efea4014b60 con 0x7efe9c0052f0 2024-04-23T16:39:32.577 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efe9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d079e76b-d697-4efc-a10e-52178c04b550", "id": 2}] 2024-04-23T16:39:32.577 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efe9b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:32.577 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 10 monclient: shutdown 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 msgr2=0x7efeb413f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efeb413f120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efea4007fd0 tx=0x7efea400e910 comp rx=0 tx=0).stop 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 shutdown_connections 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efeb41cf0c0 0x7efeb41cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.578 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeb413cf10 0x7efeb41348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 --2- 172.21.15.204:0/4094764167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efe9c0052f0 0x7efeb413f120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 >> 172.21.15.204:0/4094764167 conn(0x7efeb40a36a0 msgr2=0x7efeb40a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 shutdown_connections 2024-04-23T16:39:32.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.572+0000 7efeb90ee640 1 -- 172.21.15.204:0/4094764167 wait complete. 2024-04-23T16:39:32.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new fa572109-89d7-4f3d-bd5a-312932eb60b5 3 2024-04-23T16:39:32.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.620+0000 7f2dc5c80180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T16:39:32.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:32.684+0000 7f2dc5c80180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient: get_monmap_and_config 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient: monmap: 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:39:32.739532+0000 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:39:32.739532+0000 2024-04-23T16:39:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:32.740 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 Processor -- start 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 -- start start 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient: init 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3d740a0c00 0x7f3d740a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): picked mon.noname-a con 0x7f3d740a0c00 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3d740a0fe0 0x7f3d740a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): picked mon.noname-c con 0x7f3d740a0fe0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d7412d960 0x7f3d7412ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): picked mon.noname-b con 0x7f3d7412d960 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f3d74073a80 con 0x7f3d740a0c00 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3d740737a0 con 0x7f3d740a0fe0 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f3d740a1cb0 con 0x7f3d7412d960 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d7b799640 10 monclient(hunting): authenticate will time out at 916.849710s 2024-04-23T16:39:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3d740a0fe0 0x7f3d740a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3d740a0fe0 0x7f3d740a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56660/0 (socket says 172.21.15.204:56660) 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 -- 172.21.15.204:0/2450116280 learned_addr learned my addr 172.21.15.204:0/2450116280 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): get_auth_request con 0x7f3d740a0fe0 auth_method 0 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79535640 1 --2- 172.21.15.204:0/2450116280 >> v2:172.21.15.79:3300/0 conn(0x7f3d740a0c00 0x7f3d740a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79d36640 1 --2- 172.21.15.204:0/2450116280 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d7412d960 0x7f3d7412ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79535640 10 monclient(hunting): get_auth_request con 0x7f3d740a0c00 auth_method 0 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79535640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79535640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79d36640 10 monclient(hunting): get_auth_request con 0x7f3d7412d960 auth_method 0 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79d36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79d36640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d79d36640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 -- 172.21.15.204:0/2450116280 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d7412d960 msgr2=0x7f3d7412ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 --2- 172.21.15.204:0/2450116280 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d7412d960 0x7f3d7412ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 -- 172.21.15.204:0/2450116280 >> v2:172.21.15.79:3300/0 conn(0x7f3d740a0c00 msgr2=0x7f3d740a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 --2- 172.21.15.204:0/2450116280 >> v2:172.21.15.79:3300/0 conn(0x7f3d740a0c00 0x7f3d740a22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 monclient: found mon.noname-c 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.736+0000 7f3d78d34640 1 -- 172.21.15.204:0/2450116280 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d740a1930 con 0x7f3d740a0fe0 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d78d34640 1 -- 172.21.15.204:0/2450116280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d740a0fe0 msgr2=0x7f3d740a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d78d34640 1 --2- 172.21.15.204:0/2450116280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d740a0fe0 0x7f3d740a13f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d7406bf00 tx=0x7f3d6801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=459c63da264eb904 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 1 -- 172.21.15.204:0/2450116280 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d680b0070 con 0x7f3d740a0fe0 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: dump: 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.749 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 1 -- 172.21.15.204:0/2450116280 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d680ab600 con 0x7f3d740a0fe0 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 1 -- 172.21.15.204:0/2450116280 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d680ab9d0 con 0x7f3d740a0fe0 2024-04-23T16:39:32.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 5 monclient: authenticate success, global_id 4123 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 4 set_mon_vals no callback set 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d67fff640 10 monclient: dump: 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 10 monclient: shutdown 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 -- 172.21.15.204:0/2450116280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d740a0fe0 msgr2=0x7f3d740a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 --2- 172.21.15.204:0/2450116280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d740a0fe0 0x7f3d740a13f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d7406bf00 tx=0x7f3d6801c470 comp rx=0 tx=0).stop 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 -- 172.21.15.204:0/2450116280 shutdown_connections 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 --2- 172.21.15.204:0/2450116280 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d7412d960 0x7f3d7412ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 --2- 172.21.15.204:0/2450116280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d740a0fe0 0x7f3d740a13f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 --2- 172.21.15.204:0/2450116280 >> v2:172.21.15.79:3300/0 conn(0x7f3d740a0c00 0x7f3d740a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 -- 172.21.15.204:0/2450116280 >> 172.21.15.204:0/2450116280 conn(0x7f3d740a2d00 msgr2=0x7f3d740a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 -- 172.21.15.204:0/2450116280 shutdown_connections 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.740+0000 7f3d7b799640 1 -- 172.21.15.204:0/2450116280 wait complete. 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient: monmap: 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 Processor -- start 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 -- start start 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient: init 2024-04-23T16:39:32.753 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d740a0c00 0x7f3d741cf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): picked mon.noname-a con 0x7f3d740a0c00 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d740a0fe0 0x7f3d7413d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): picked mon.noname-c con 0x7f3d740a0fe0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 0x7f3d7413ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): picked mon.noname-b con 0x7f3d7412d960 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3d740a2970 con 0x7f3d740a0c00 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3d740a27f0 con 0x7f3d7412d960 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f3d74141290 con 0x7f3d740a0fe0 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.754 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d7b799640 10 monclient(hunting): authenticate will time out at 916.856977s 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 0x7f3d7413ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 0x7f3d7413ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56672/0 (socket says 172.21.15.204:56672) 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 -- 172.21.15.204:0/3454534593 learned_addr learned my addr 172.21.15.204:0/3454534593 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): get_auth_request con 0x7f3d7412d960 auth_method 0 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79535640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d740a0c00 0x7f3d741cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d78d34640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d740a0fe0 0x7f3d7413d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79535640 10 monclient(hunting): get_auth_request con 0x7f3d740a0c00 auth_method 0 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79535640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79535640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d78d34640 10 monclient(hunting): get_auth_request con 0x7f3d740a0fe0 auth_method 0 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d78d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d78d34640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d78d34640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 -- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d740a0fe0 msgr2=0x7f3d7413d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d740a0fe0 0x7f3d7413d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 -- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d740a0c00 msgr2=0x7f3d741cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d740a0c00 0x7f3d741cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 monclient: found mon.noname-b 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 -- 172.21.15.204:0/3454534593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7409cc50 con 0x7f3d7412d960 2024-04-23T16:39:32.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 -- 172.21.15.204:0/3454534593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 msgr2=0x7f3d7413ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d79d36640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 0x7f3d7413ddb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c007e40 tx=0x7f3d6c00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c013790 con 0x7f3d7412d960 2024-04-23T16:39:32.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:32.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d65ffb640 10 monclient: dump: 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.744+0000 7f3d65ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.759 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 5 monclient: authenticate success, global_id 4126 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d6c019070 con 0x7f3d7412d960 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 monclient: _renew_subs 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d74138b20 con 0x7f3d7412d960 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c013b90 con 0x7f3d7412d960 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: dump: 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.760 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d677fe640 4 set_mon_vals no callback set 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 monclient: _renew_subs 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7413e2f0 con 0x7f3d7412d960 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3d6c01e460 con 0x7f3d7412d960 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f3d6c0149a0 con 0x7f3d7412d960 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.762 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d74068ae0 con 0x7f3d7412d960 2024-04-23T16:39:32.762 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3d6c014cc0 con 0x7f3d7412d960 2024-04-23T16:39:32.762 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.762 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.748+0000 7f3d65ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:32.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.816+0000 7f3d7b799640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fa572109-89d7-4f3d-bd5a-312932eb60b5", "id": 3}] 2024-04-23T16:39:32.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.816+0000 7f3d7b799640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fa572109-89d7-4f3d-bd5a-312932eb60b5", "id": 3}] 2024-04-23T16:39:32.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.816+0000 7f3d7b799640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.816+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fa572109-89d7-4f3d-bd5a-312932eb60b5", "id": 3} v 0) -- 0x7f3d7406c210 con 0x7f3d7412d960 2024-04-23T16:39:32.828 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d65ffb640 1 -- 172.21.15.204:0/3454534593 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa572109-89d7-4f3d-bd5a-312932eb60b5", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f3d6c014510 con 0x7f3d7412d960 2024-04-23T16:39:32.828 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fa572109-89d7-4f3d-bd5a-312932eb60b5", "id": 3}] 2024-04-23T16:39:32.828 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d65ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:32.828 INFO:teuthology.orchestra.run.smithi204.stdout:3 2024-04-23T16:39:32.829 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 10 monclient: shutdown 2024-04-23T16:39:32.829 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.829 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 msgr2=0x7f3d7413ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.829 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 0x7f3d7413ddb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c007e40 tx=0x7f3d6c00e4b0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 shutdown_connections 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7412d960 0x7f3d7413ddb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3d740a0fe0 0x7f3d7413d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 --2- 172.21.15.204:0/3454534593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d740a0c00 0x7f3d741cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 >> 172.21.15.204:0/3454534593 conn(0x7f3d740a2d00 msgr2=0x7f3d74130540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 shutdown_connections 2024-04-23T16:39:32.830 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.824+0000 7f3d7b799640 1 -- 172.21.15.204:0/3454534593 wait complete. 2024-04-23T16:39:32.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new b34e743a-ccc5-46b6-8512-69cd9d149b44 4 2024-04-23T16:39:32.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient: get_monmap_and_config 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient: monmap: 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:39:32.956157+0000 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:39:32.956157+0000 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:32.956 INFO:teuthology.orchestra.run.smithi204.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:32.957 INFO:teuthology.orchestra.run.smithi204.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:32.957 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 Processor -- start 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 -- start start 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient: init 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6c01355c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fa6c009c690 0x7fa6c013a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c009c690 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa6c013aa80 0x7fa6c013ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c013aa80 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6c009bdc0 con 0x7fa6c01355c0 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fa6c0072bb0 con 0x7fa6c009c690 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa6c009f6c0 con 0x7fa6c013aa80 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c7633640 10 monclient(hunting): authenticate will time out at 917.066238s 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa6c013aa80 0x7fa6c013ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c53cf640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa6c01355c0 0x7fa6c01359a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:58684/0 (socket says 172.21.15.204:58684) 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c53cf640 1 -- 172.21.15.204:0/1404718744 learned_addr learned my addr 172.21.15.204:0/1404718744 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): get_auth_request con 0x7fa6c013aa80 auth_method 0 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c4bce640 1 --2- 172.21.15.204:0/1404718744 >> v2:172.21.15.79:3300/0 conn(0x7fa6c009c690 0x7fa6c013a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c4bce640 10 monclient(hunting): get_auth_request con 0x7fa6c009c690 auth_method 0 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c4bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c4bce640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c4bce640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/1404718744 >> v2:172.21.15.79:3300/0 conn(0x7fa6c009c690 msgr2=0x7fa6c013a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 1 --2- 172.21.15.204:0/1404718744 >> v2:172.21.15.79:3300/0 conn(0x7fa6c009c690 0x7fa6c013a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/1404718744 >> v1:172.21.15.204:6789/0 conn(0x7fa6c01355c0 legacy=0x7fa6c01359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 1 monclient: found mon.noname-c 2024-04-23T16:39:32.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.952+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/1404718744 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c009f1e0 con 0x7fa6c013aa80 2024-04-23T16:39:32.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/1404718744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 msgr2=0x7fa6c013ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c5bd0640 1 --2- 172.21.15.204:0/1404718744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c013ce70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa6c009f9d0 tx=0x7fa6b801c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c5fc6f9d6a586a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 1 -- 172.21.15.204:0/1404718744 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80b2070 con 0x7fa6c013aa80 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: dump: 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 1 -- 172.21.15.204:0/1404718744 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6b80a5810 con 0x7fa6c013aa80 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 5 monclient: authenticate success, global_id 4129 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 1 -- 172.21.15.204:0/1404718744 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80a5be0 con 0x7fa6c013aa80 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6b7fff640 10 monclient: dump: 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 4 set_mon_vals no callback set 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 10 monclient: shutdown 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:32.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 -- 172.21.15.204:0/1404718744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 msgr2=0x7fa6c013ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 --2- 172.21.15.204:0/1404718744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c013ce70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa6c009f9d0 tx=0x7fa6b801c840 comp rx=0 tx=0).stop 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 -- 172.21.15.204:0/1404718744 shutdown_connections 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 --2- 172.21.15.204:0/1404718744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c013ce70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 --2- 172.21.15.204:0/1404718744 >> v2:172.21.15.79:3300/0 conn(0x7fa6c009c690 0x7fa6c013a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 -- 172.21.15.204:0/1404718744 >> 172.21.15.204:0/1404718744 conn(0x7fa6c00a6250 msgr2=0x7fa6c00a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 -- 172.21.15.204:0/1404718744 shutdown_connections 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.956+0000 7fa6c7633640 1 -- 172.21.15.204:0/1404718744 wait complete. 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient: build_initial_monmap 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient: monmap: 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:32.967 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 Processor -- start 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 -- start start 2024-04-23T16:39:32.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient: init 2024-04-23T16:39:32.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:32.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:32.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6c009c690 0x7fa6c0131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c009c690 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6c01355c0 0x7fa6c006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c01355c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c013aa80 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c013fd50 con 0x7fa6c009c690 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c013fbd0 con 0x7fa6c013aa80 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c013fed0 con 0x7fa6c01355c0 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c7633640 10 monclient(hunting): authenticate will time out at 917.073693s 2024-04-23T16:39:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c006e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56692/0 (socket says 172.21.15.204:56692) 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/3430538935 learned_addr learned my addr 172.21.15.204:0/3430538935 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): get_auth_request con 0x7fa6c013aa80 auth_method 0 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c53cf640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6c009c690 0x7fa6c0131320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c4bce640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6c01355c0 0x7fa6c006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6c01355c0 msgr2=0x7fa6c006de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6c01355c0 0x7fa6c006de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6c009c690 msgr2=0x7fa6c0131320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6c009c690 0x7fa6c0131320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient: _finish_hunting 0 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 monclient: found mon.noname-b 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/3430538935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c00719a0 con 0x7fa6c013aa80 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c53cf640 10 monclient: get_auth_request con 0x7fa6c009c690 auth_method 0 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.960+0000 7fa6c53cf640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6c009c690 0x7fa6c0131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c5bd0640 1 -- 172.21.15.204:0/3430538935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 msgr2=0x7fa6c006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c5bd0640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c006e3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa6b801a470 tx=0x7fa6b80195e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80b2050 con 0x7fa6c013aa80 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: dump: 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 5 monclient: authenticate success, global_id 4132 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 monclient: _renew_subs 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6b80be070 con 0x7fa6c013aa80 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c0072810 con 0x7fa6c013aa80 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80adc80 con 0x7fa6c013aa80 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: dump: 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:32.976 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 monclient: _renew_subs 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b77fe640 4 set_mon_vals no callback set 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c00716c0 con 0x7fa6c013aa80 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa6b80c3460 con 0x7fa6c013aa80 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fa6b80b9960 con 0x7fa6c013aa80 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c00692e0 con 0x7fa6c013aa80 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa6b80164c0 con 0x7fa6c013aa80 2024-04-23T16:39:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:32.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:32.964+0000 7fa6b5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:33.084 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.080+0000 7fa6c7633640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b34e743a-ccc5-46b6-8512-69cd9d149b44", "id": 4}] 2024-04-23T16:39:33.084 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.080+0000 7fa6c7633640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b34e743a-ccc5-46b6-8512-69cd9d149b44", "id": 4}] 2024-04-23T16:39:33.084 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.080+0000 7fa6c7633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:33.084 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.080+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b34e743a-ccc5-46b6-8512-69cd9d149b44", "id": 4} v 0) -- 0x7fa6c00008e0 con 0x7fa6c013aa80 2024-04-23T16:39:33.090 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.084+0000 7fa6b5ffb640 1 -- 172.21.15.204:0/3430538935 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b34e743a-ccc5-46b6-8512-69cd9d149b44", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fa6b80b9450 con 0x7fa6c013aa80 2024-04-23T16:39:33.090 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.084+0000 7fa6b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b34e743a-ccc5-46b6-8512-69cd9d149b44", "id": 4}] 2024-04-23T16:39:33.090 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.084+0000 7fa6b5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:33.091 INFO:teuthology.orchestra.run.smithi204.stdout:4 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 10 monclient: shutdown 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 msgr2=0x7fa6c006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c006e3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa6b801a470 tx=0x7fa6b80195e0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 shutdown_connections 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6c013aa80 0x7fa6c006e3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6c01355c0 0x7fa6c006de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 --2- 172.21.15.204:0/3430538935 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6c009c690 0x7fa6c0131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 >> 172.21.15.204:0/3430538935 conn(0x7fa6c00a6250 msgr2=0x7fa6c00a1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 shutdown_connections 2024-04-23T16:39:33.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.088+0000 7fa6c7633640 1 -- 172.21.15.204:0/3430538935 wait complete. 2024-04-23T16:39:33.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5 5 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient: get_monmap_and_config 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient: build_initial_monmap 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient: monmap: 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:39:33.217087+0000 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:39:33.217087+0000 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:33.217 INFO:teuthology.orchestra.run.smithi204.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:33.218 INFO:teuthology.orchestra.run.smithi204.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:33.218 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 Processor -- start 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 -- start start 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient: init 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a90137ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): picked mon.noname-b con 0x7f4a9009c290 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a9009c670 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f4a9009ca70 0x7f4a900a2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): picked mon.noname-a con 0x7f4a9009ca70 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:33.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a90073360 con 0x7f4a9009c670 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f4a9009d150 con 0x7f4a9009ca70 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4a9009a290 con 0x7f4a9009c290 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a974b7640 10 monclient(hunting): authenticate will time out at 917.327830s 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a94a52640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f4a9009c670 0x7f4a900a1b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:58688/0 (socket says 172.21.15.204:58688) 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.212+0000 7f4a94a52640 1 -- 172.21.15.204:0/2211160153 learned_addr learned my addr 172.21.15.204:0/2211160153 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2211160153 >> v2:172.21.15.79:3300/0 conn(0x7f4a9009ca70 0x7f4a900a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a90137ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): get_auth_request con 0x7f4a9009ca70 auth_method 0 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): get_auth_request con 0x7f4a9009c290 auth_method 0 2024-04-23T16:39:33.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 1 -- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 msgr2=0x7f4a90137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a90137ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 1 -- 172.21.15.204:0/2211160153 >> v1:172.21.15.204:6789/0 conn(0x7f4a9009c670 legacy=0x7f4a900a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient: _finish_hunting 0 2024-04-23T16:39:33.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 1 monclient: found mon.noname-a 2024-04-23T16:39:33.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:33.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:33.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95a54640 1 -- 172.21.15.204:0/2211160153 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9009a570 con 0x7f4a9009ca70 2024-04-23T16:39:33.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.216+0000 7f4a95253640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a90137ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 1 -- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009ca70 msgr2=0x7f4a900a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009ca70 0x7f4a900a2280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a90067e20 tx=0x7f4a8801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cf1c2fff4b0cdf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 1 -- 172.21.15.204:0/2211160153 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a880b2070 con 0x7f4a9009ca70 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: dump: 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 1 -- 172.21.15.204:0/2211160153 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a880a5820 con 0x7f4a9009ca70 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 5 monclient: authenticate success, global_id 4125 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 1 -- 172.21.15.204:0/2211160153 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a880a5bf0 con 0x7f4a9009ca70 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a87fff640 10 monclient: dump: 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:33.226 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:33.227 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 4 set_mon_vals no callback set 2024-04-23T16:39:33.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: shutdown 2024-04-23T16:39:33.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:33.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009ca70 msgr2=0x7f4a900a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:33.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009ca70 0x7f4a900a2280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a90067e20 tx=0x7f4a8801c780 comp rx=0 tx=0).stop 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- 172.21.15.204:0/2211160153 shutdown_connections 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009ca70 0x7f4a900a2280 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2211160153 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a90137ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- 172.21.15.204:0/2211160153 >> 172.21.15.204:0/2211160153 conn(0x7f4a900a2c60 msgr2=0x7f4a900a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- 172.21.15.204:0/2211160153 shutdown_connections 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- 172.21.15.204:0/2211160153 wait complete. 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: build_initial_monmap 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: monmap: 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:33.232 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 Processor -- start 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- start start 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: init 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a901cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): picked mon.noname-c con 0x7f4a9009c290 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009c670 0x7f4a901cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): picked mon.noname-a con 0x7f4a9009c670 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): picked mon.noname-b con 0x7f4a9009ca70 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4a900a2940 con 0x7f4a9009c670 2024-04-23T16:39:33.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4a900a27c0 con 0x7f4a9009ca70 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4a901421c0 con 0x7f4a9009c290 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a974b7640 10 monclient(hunting): authenticate will time out at 917.335850s 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56694/0 (socket says 172.21.15.204:56694) 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 1 -- 172.21.15.204:0/2837198119 learned_addr learned my addr 172.21.15.204:0/2837198119 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 10 monclient(hunting): get_auth_request con 0x7f4a9009ca70 auth_method 0 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.220+0000 7f4a95a54640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:33.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a94a52640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009c670 0x7f4a901cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95253640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a901cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 -- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 msgr2=0x7f4a901cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a901cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 -- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009c670 msgr2=0x7f4a901cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009c670 0x7f4a901cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 10 monclient: _finish_hunting 0 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 monclient: found mon.noname-b 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 -- 172.21.15.204:0/2837198119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a90136e40 con 0x7f4a9009ca70 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 -- 172.21.15.204:0/2837198119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 msgr2=0x7f4a901cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:33.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a8801d020 tx=0x7f4a88019520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a880bb020 con 0x7f4a9009ca70 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: dump: 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:33.236 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 5 monclient: authenticate success, global_id 4135 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a880ad850 con 0x7f4a9009ca70 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 monclient: _renew_subs 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a90073030 con 0x7f4a9009ca70 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 monclient: _renew_subs 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a880b2040 con 0x7f4a9009ca70 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:33.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9012c2b0 con 0x7f4a9009ca70 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a877fe640 4 set_mon_vals no callback set 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 10 monclient: dump: 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:33.238 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4a880be020 con 0x7f4a9009ca70 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f4a880b7990 con 0x7f4a9009ca70 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.224+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a90068ae0 con 0x7f4a9009ca70 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.228+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4a880c5030 con 0x7f4a9009ca70 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.228+0000 7f4a85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.228+0000 7f4a85ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:33.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-23T16:39:33.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-23T16:39:33.279 INFO:tasks.ceph.mgr.x.smithi204.stderr: from numpy import show_config as show_numpy_config 2024-04-23T16:39:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.292+0000 7f4a974b7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5", "id": 5}] 2024-04-23T16:39:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.292+0000 7f4a974b7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5", "id": 5}] 2024-04-23T16:39:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.292+0000 7f4a974b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.292+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5", "id": 5} v 0) -- 0x7f4a9013e580 con 0x7f4a9009ca70 2024-04-23T16:39:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:33.300+0000 7f4a95a54640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a8801d020 tx=0x7f4a88019520 comp rx=0 tx=0).queue_received will delay after 1.71389e+09 on 0x7f4a90136e40 mon_command_ack([{"prefix": "osd new", "uuid": "8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5", "id": 5}]=0 v8) 2024-04-23T16:39:33.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:33.364+0000 7f2dc5c80180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T16:39:33.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:33.544+0000 7f2dc5c80180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:39:33.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:33.720+0000 7f2dc5c80180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T16:39:33.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:33.792+0000 7f2dc5c80180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T16:39:33.944 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:33.940+0000 7f2dc5c80180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:39:34.133 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a85ffb640 1 -- 172.21.15.204:0/2837198119 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f4a90136e40 con 0x7f4a9009ca70 2024-04-23T16:39:34.133 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5", "id": 5}] 2024-04-23T16:39:34.133 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a85ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:34.133 INFO:teuthology.orchestra.run.smithi204.stdout:5 2024-04-23T16:39:34.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 10 monclient: shutdown 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 msgr2=0x7f4a901cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a8801d020 tx=0x7f4a88019520 comp rx=0 tx=0).stop 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 shutdown_connections 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a9009ca70 0x7f4a901cfdc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4a9009c670 0x7f4a901cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 --2- 172.21.15.204:0/2837198119 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4a9009c290 0x7f4a901cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 >> 172.21.15.204:0/2837198119 conn(0x7f4a900a2c60 msgr2=0x7f4a90135af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 shutdown_connections 2024-04-23T16:39:34.135 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T16:39:34.128+0000 7f4a974b7640 1 -- 172.21.15.204:0/2837198119 wait complete. 2024-04-23T16:39:34.149 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-23T16:39:34.150 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-23T16:39:34.152 INFO:tasks.ceph.osd.0:Started 2024-04-23T16:39:34.152 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-23T16:39:34.153 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-23T16:39:34.155 INFO:tasks.ceph.osd.1:Started 2024-04-23T16:39:34.155 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-23T16:39:34.155 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-23T16:39:34.157 INFO:tasks.ceph.osd.2:Started 2024-04-23T16:39:34.158 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-23T16:39:34.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-23T16:39:34.160 INFO:tasks.ceph.osd.3:Started 2024-04-23T16:39:34.160 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-23T16:39:34.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-23T16:39:34.162 INFO:tasks.ceph.osd.4:Started 2024-04-23T16:39:34.162 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-23T16:39:34.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-23T16:39:34.164 INFO:tasks.ceph.osd.5:Started 2024-04-23T16:39:34.164 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient: get_monmap_and_config 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient: build_initial_monmap 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient: monmap: 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:34.285776+0000 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:34.285776+0000 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:34.284 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:34.285 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:34.285 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:34.285 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:34.285 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 Processor -- start 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 -- start start 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient: init 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc3cc06ca30 0x7fc3cc0a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): picked mon.noname-c con 0x7fc3cc06ca30 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fc3cc0a3370 0x7fc3cc0a3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:34.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): picked mon.noname-a con 0x7fc3cc0a3370 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3cc0a47e0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3cc0a2f20 con 0x7fc3cc0a47e0 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fc3cc0a1b50 con 0x7fc3cc0a3370 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc3cc06ce10 con 0x7fc3cc06ca30 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3d0c8a640 10 monclient(hunting): authenticate will time out at 928.473462s 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fc3cc0a3370 0x7fc3cc0a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fc3cc0a3370 0x7fc3cc0a3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45382/0 (socket says 172.21.15.79:45382) 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 -- 172.21.15.79:0/3164633376 learned_addr learned my addr 172.21.15.79:0/3164633376 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): get_auth_request con 0x7fc3cc0a3370 auth_method 0 2024-04-23T16:39:34.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 1 --2- 172.21.15.79:0/3164633376 >> v2:172.21.15.204:3300/0 conn(0x7fc3cc06ca30 0x7fc3cc0a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): get_auth_request con 0x7fc3cc06ca30 auth_method 0 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:34.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 -- 172.21.15.79:0/3164633376 >> v2:172.21.15.204:3300/0 conn(0x7fc3cc06ca30 msgr2=0x7fc3cc0a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 --2- 172.21.15.79:0/3164633376 >> v2:172.21.15.204:3300/0 conn(0x7fc3cc06ca30 0x7fc3cc0a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 -- 172.21.15.79:0/3164633376 >> v1:172.21.15.204:6789/0 conn(0x7fc3cc0a47e0 legacy=0x7fc3cc0a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient: _finish_hunting 0 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 monclient: found mon.noname-a 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:34.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.282+0000 7fc3c9d9b640 1 -- 172.21.15.79:0/3164633376 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3cc0a5860 con 0x7fc3cc0a3370 2024-04-23T16:39:34.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c9d9b640 1 -- 172.21.15.79:0/3164633376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc0a3370 msgr2=0x7fc3cc0a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c9d9b640 1 --2- 172.21.15.79:0/3164633376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc0a3370 0x7fc3cc0a3780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0a7780 tx=0x7fc3c001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e62bdb6afa197d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 1 -- 172.21.15.79:0/3164633376 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c00b0070 con 0x7fc3cc0a3370 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: dump: 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:34.292 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: _finish_auth 0 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 1 -- 172.21.15.79:0/3164633376 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3c00ab600 con 0x7fc3cc0a3370 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 1 -- 172.21.15.79:0/3164633376 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c00ab9d0 con 0x7fc3cc0a3370 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 5 monclient: authenticate success, global_id 4143 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 4 set_mon_vals no callback set 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3c959a640 10 monclient: dump: 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.293 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 10 monclient: shutdown 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/3164633376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc0a3370 msgr2=0x7fc3cc0a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/3164633376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc0a3370 0x7fc3cc0a3780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0a7780 tx=0x7fc3c001c430 comp rx=0 tx=0).stop 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/3164633376 shutdown_connections 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/3164633376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc0a3370 0x7fc3cc0a3780 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/3164633376 >> v2:172.21.15.204:3300/0 conn(0x7fc3cc06ca30 0x7fc3cc0a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/3164633376 >> 172.21.15.79:0/3164633376 conn(0x7fc3cc0aa3e0 msgr2=0x7fc3cc0a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:34.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/3164633376 shutdown_connections 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/3164633376 wait complete. 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 10 monclient: build_initial_monmap 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.286+0000 7fc3d0c8a640 10 monclient: monmap: 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 Processor -- start 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 -- start start 2024-04-23T16:39:34.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: init 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 0x7fc3cc1cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): picked mon.noname-a con 0x7fc3cc06ca30 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc3cc0a3370 0x7fc3cc13b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): picked mon.noname-c con 0x7fc3cc0a3370 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3cc0a47e0 0x7fc3cc122250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): picked mon.noname-b con 0x7fc3cc0a47e0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc142490 con 0x7fc3cc06ca30 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc142310 con 0x7fc3cc0a47e0 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:34.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc3cc142610 con 0x7fc3cc0a3370 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient(hunting): authenticate will time out at 928.480578s 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 0x7fc3cc1cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 0x7fc3cc1cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45384/0 (socket says 172.21.15.79:45384) 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 -- 172.21.15.79:0/408934771 learned_addr learned my addr 172.21.15.79:0/408934771 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): get_auth_request con 0x7fc3cc06ca30 auth_method 0 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc3cc0a3370 0x7fc3cc13b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): get_auth_request con 0x7fc3cc0a3370 auth_method 0 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:34.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3cad9d640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3cc0a47e0 0x7fc3cc122250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3cad9d640 10 monclient(hunting): get_auth_request con 0x7fc3cc0a47e0 auth_method 0 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3cad9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 -- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc3cc0a3370 msgr2=0x7fc3cc13b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc3cc0a3370 0x7fc3cc13b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 -- 172.21.15.79:0/408934771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3cc0a47e0 msgr2=0x7fc3cc122250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3cc0a47e0 0x7fc3cc122250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 monclient: found mon.noname-a 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 -- 172.21.15.79:0/408934771 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3cc0a01f0 con 0x7fc3cc06ca30 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3cad9d640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3cc0a47e0 0x7fc3cc122250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 -- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 msgr2=0x7fc3cc1cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3ca59c640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 0x7fc3cc1cf910 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00efc0 tx=0x7fc3bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc018020 con 0x7fc3cc06ca30 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: dump: 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:34.302 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3bc014680 con 0x7fc3cc06ca30 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc014a70 con 0x7fc3cc06ca30 2024-04-23T16:39:34.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 10 monclient: dump: 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 5 monclient: authenticate success, global_id 4146 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: _renew_subs 2024-04-23T16:39:34.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3cc125780 con 0x7fc3cc06ca30 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3c8d99640 4 set_mon_vals no callback set 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: _renew_subs 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3cc123d00 con 0x7fc3cc06ca30 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc3bc01b020 con 0x7fc3cc06ca30 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.290+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fc3bc013af0 con 0x7fc3cc06ca30 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.294+0000 7fc3d0c8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.294+0000 7fc3d0c8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.294+0000 7fc3d0c8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:34.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.294+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3cc068ae0 con 0x7fc3cc06ca30 2024-04-23T16:39:34.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.306+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc3bc022070 con 0x7fc3cc06ca30 2024-04-23T16:39:34.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.306+0000 7fc3baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:34.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.306+0000 7fc3baffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.362+0000 7fc3d0c8a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.362+0000 7fc3d0c8a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.362+0000 7fc3d0c8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.362+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3cc072040 con 0x7fc3cc06ca30 2024-04-23T16:39:34.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3baffd640 1 -- 172.21.15.79:0/408934771 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fc3bc010040 con 0x7fc3cc06ca30 2024-04-23T16:39:34.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:34.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3baffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:34.368 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:34.368 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":8,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:39:33.300300+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:39:34.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 10 monclient: shutdown 2024-04-23T16:39:34.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:34.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 msgr2=0x7fc3cc1cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 0x7fc3cc1cf910 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00efc0 tx=0x7fc3bc00c490 comp rx=0 tx=0).stop 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 shutdown_connections 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3cc0a47e0 0x7fc3cc122250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc3cc0a3370 0x7fc3cc13b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 --2- 172.21.15.79:0/408934771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3cc06ca30 0x7fc3cc1cf910 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 >> 172.21.15.79:0/408934771 conn(0x7fc3cc0aa3e0 msgr2=0x7fc3cc09e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 shutdown_connections 2024-04-23T16:39:34.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:34.366+0000 7fc3d0c8a640 1 -- 172.21.15.79:0/408934771 wait complete. 2024-04-23T16:39:34.385 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-23T16:39:34.385 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-23T16:39:34.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:34.568+0000 7f2dc5c80180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T16:39:34.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:34.644+0000 7f2dc5c80180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T16:39:34.768 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-04-23T16:39:34.764+0000 7f52187297c0 -1 Falling back to public interface 2024-04-23T16:39:34.778 INFO:tasks.ceph.osd.2.smithi079.stderr:2024-04-23T16:39:34.774+0000 7fa5994707c0 -1 Falling back to public interface 2024-04-23T16:39:34.778 INFO:tasks.ceph.osd.0.smithi079.stderr:2024-04-23T16:39:34.774+0000 7f4cfafff7c0 -1 Falling back to public interface 2024-04-23T16:39:34.778 INFO:tasks.ceph.osd.1.smithi079.stderr:2024-04-23T16:39:34.774+0000 7f5dc87fa7c0 -1 Falling back to public interface 2024-04-23T16:39:34.784 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-04-23T16:39:34.780+0000 7f4ce793b7c0 -1 Falling back to public interface 2024-04-23T16:39:34.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-04-23T16:39:34.784+0000 7f9adc7147c0 -1 Falling back to public interface 2024-04-23T16:39:34.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:34.828+0000 7f2dc5c80180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T16:39:34.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:34.904+0000 7f2dc5c80180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T16:39:34.989 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T16:39:35.107 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.102+0000 7f75c04a1640 10 monclient: get_monmap_and_config 2024-04-23T16:39:35.107 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.102+0000 7f75c04a1640 10 monclient: build_initial_monmap 2024-04-23T16:39:35.107 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.102+0000 7f75c04a1640 10 monclient: monmap: 2024-04-23T16:39:35.107 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-04-23T16:39:35.107 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:35.107 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:39:35.109259+0000 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:39:35.109259+0000 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:35.108 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:35.110 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 Processor -- start 2024-04-23T16:39:35.110 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 -- start start 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient: init 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75b80a36a0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f75b80a1bc0 0x7f75b8139610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): picked mon.noname-c con 0x7f75b80a1bc0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f75b8139b50 0x7f75b813bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): picked mon.noname-a con 0x7f75b8139b50 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b80a2350 con 0x7f75b80a36a0 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f75b80a3a80 con 0x7f75b8139b50 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:35.111 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f75b80a25c0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75c04a1640 10 monclient(hunting): authenticate will time out at 929.297441s 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f75b8139b50 0x7f75b813bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f75b8139b50 0x7f75b813bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45400/0 (socket says 172.21.15.79:45400) 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 1 -- 172.21.15.79:0/4061403193 learned_addr learned my addr 172.21.15.79:0/4061403193 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 10 monclient(hunting): get_auth_request con 0x7f75b8139b50 auth_method 0 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bea3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75be23d640 1 --1- 172.21.15.79:0/4061403193 >> v1:172.21.15.79:6789/0 conn(0x7f75b80a36a0 0x7f75b809fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:38500/0 (socket says 172.21.15.79:38500) 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 1 --2- 172.21.15.79:0/4061403193 >> v2:172.21.15.204:3300/0 conn(0x7f75b80a1bc0 0x7f75b8139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): get_auth_request con 0x7f75b80a1bc0 auth_method 0 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:35.112 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 1 -- 172.21.15.79:0/4061403193 >> v2:172.21.15.79:3300/0 conn(0x7f75b8139b50 msgr2=0x7f75b813bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 1 --2- 172.21.15.79:0/4061403193 >> v2:172.21.15.79:3300/0 conn(0x7f75b8139b50 0x7f75b813bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 1 -- 172.21.15.79:0/4061403193 >> v1:172.21.15.79:6789/0 conn(0x7f75b80a36a0 legacy=0x7f75b809fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 1 monclient: found mon.noname-c 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:35.113 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.106+0000 7f75bda3c640 1 -- 172.21.15.79:0/4061403193 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b80a02a0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.114 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bda3c640 1 -- 172.21.15.79:0/4061403193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a1bc0 msgr2=0x7f75b8139610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:35.114 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bda3c640 1 --2- 172.21.15.79:0/4061403193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a1bc0 0x7f75b8139610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75b812e9b0 tx=0x7f75a801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=903c4cb90cb337f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 1 -- 172.21.15.79:0/4061403193 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a80b0070 con 0x7f75b80a1bc0 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: dump: 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:35.116 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: _finish_auth 0 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: _check_auth_tickets 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 1 -- 172.21.15.79:0/4061403193 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a80ab600 con 0x7f75b80a1bc0 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 1 -- 172.21.15.79:0/4061403193 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a80ab9d0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75bd23b640 10 monclient: dump: 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:35.117 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75c04a1640 5 monclient: authenticate success, global_id 4156 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75c04a1640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:35.118 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.110+0000 7f75c04a1640 4 set_mon_vals no callback set 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient: shutdown 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- 172.21.15.79:0/4061403193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a1bc0 msgr2=0x7f75b8139610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 --2- 172.21.15.79:0/4061403193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a1bc0 0x7f75b8139610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75b812e9b0 tx=0x7f75a801c300 comp rx=0 tx=0).stop 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- 172.21.15.79:0/4061403193 shutdown_connections 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 --2- 172.21.15.79:0/4061403193 >> v2:172.21.15.79:3300/0 conn(0x7f75b8139b50 0x7f75b813bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 --2- 172.21.15.79:0/4061403193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a1bc0 0x7f75b8139610 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- 172.21.15.79:0/4061403193 >> 172.21.15.79:0/4061403193 conn(0x7f75b80a2bb0 msgr2=0x7f75b80a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- 172.21.15.79:0/4061403193 shutdown_connections 2024-04-23T16:39:35.123 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- 172.21.15.79:0/4061403193 wait complete. 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient: build_initial_monmap 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient: monmap: 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 Processor -- start 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- start start 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient: init 2024-04-23T16:39:35.124 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 0x7f75b812db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): picked mon.noname-a con 0x7f75b80a1bc0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a36a0 0x7f75b812e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): picked mon.noname-b con 0x7f75b80a36a0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f75b8139b50 0x7f75b81320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): picked mon.noname-c con 0x7f75b8139b50 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f75b813f030 con 0x7f75b80a1bc0 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f75b813eeb0 con 0x7f75b80a36a0 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f75b813f1b0 con 0x7f75b8139b50 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:35.125 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75c04a1640 10 monclient(hunting): authenticate will time out at 929.306083s 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 0x7f75b812db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f75b8139b50 0x7f75b81320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 0x7f75b812db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45416/0 (socket says 172.21.15.79:45416) 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 -- 172.21.15.79:0/259881103 learned_addr learned my addr 172.21.15.79:0/259881103 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): get_auth_request con 0x7f75b80a1bc0 auth_method 0 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): get_auth_request con 0x7f75b8139b50 auth_method 0 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a36a0 0x7f75b812e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:35.126 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): get_auth_request con 0x7f75b80a36a0 auth_method 0 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bda3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2024-04-23T16:39:35.127 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 -- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f75b8139b50 msgr2=0x7f75b81320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f75b8139b50 0x7f75b81320f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 -- 172.21.15.79:0/259881103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a36a0 msgr2=0x7f75b812e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a36a0 0x7f75b812e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 monclient: found mon.noname-a 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75be23d640 1 -- 172.21.15.79:0/259881103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b8068490 con 0x7f75b80a1bc0 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.114+0000 7f75bea3e640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f75b8139b50 0x7f75b81320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75be23d640 1 -- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 msgr2=0x7f75b812db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75be23d640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 0x7f75b812db60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75b400c460 tx=0x7f75b400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4017070 con 0x7f75b80a1bc0 2024-04-23T16:39:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: dump: 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:35.129 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b4005820 con 0x7f75b80a1bc0 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4005c10 con 0x7f75b80a1bc0 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 5 monclient: authenticate success, global_id 4152 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 monclient: _renew_subs 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75affff640 4 set_mon_vals no callback set 2024-04-23T16:39:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b8133ab0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 10 monclient: dump: 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 monclient: _renew_subs 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b812e5e0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75b401a070 con 0x7f75b80a1bc0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f75b40129f0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.118+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b8068f20 con 0x7f75b80a1bc0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.122+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f75b4021020 con 0x7f75b80a1bc0 2024-04-23T16:39:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.122+0000 7f75aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:35.133 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.122+0000 7f75aeffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:35.181 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:35.181 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:35.181 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:35.181 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75b8137fa0 con 0x7f75b80a1bc0 2024-04-23T16:39:35.182 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75aeffd640 1 -- 172.21.15.79:0/259881103 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f75b8068490 con 0x7f75b80a1bc0 2024-04-23T16:39:35.182 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:35.182 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75aeffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:35.182 INFO:teuthology.misc.health.smithi079.stdout: 2024-04-23T16:39:35.183 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":8,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:39:33.300300+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:39:35.183 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 10 monclient: shutdown 2024-04-23T16:39:35.183 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:35.183 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 msgr2=0x7f75b812db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 0x7f75b812db60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75b400c460 tx=0x7f75b400c920 comp rx=0 tx=0).stop 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 shutdown_connections 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f75b8139b50 0x7f75b81320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b80a36a0 0x7f75b812e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 --2- 172.21.15.79:0/259881103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b80a1bc0 0x7f75b812db60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 >> 172.21.15.79:0/259881103 conn(0x7f75b80a2bb0 msgr2=0x7f75b80a4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 shutdown_connections 2024-04-23T16:39:35.184 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:35.178+0000 7f75c04a1640 1 -- 172.21.15.79:0/259881103 wait complete. 2024-04-23T16:39:35.197 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-23T16:39:35.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:35.192+0000 7f2dc5c80180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T16:39:35.260 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:35.256+0000 7f2dc5c80180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T16:39:36.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:36.652+0000 7f2d6d0c9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-23T16:39:36.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-04-23T16:39:36.652+0000 7f2d6d0c9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-23T16:39:38.517 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-04-23T16:39:38.512+0000 7f52187297c0 -1 osd.3 0 log_to_monitors true 2024-04-23T16:39:38.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-04-23T16:39:38.592+0000 7f9adc7147c0 -1 osd.5 0 log_to_monitors true 2024-04-23T16:39:38.674 INFO:tasks.ceph.osd.2.smithi079.stderr:2024-04-23T16:39:38.670+0000 7fa5994707c0 -1 osd.2 0 log_to_monitors true 2024-04-23T16:39:38.680 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-04-23T16:39:38.676+0000 7f4ce793b7c0 -1 osd.4 0 log_to_monitors true 2024-04-23T16:39:38.698 INFO:tasks.ceph.osd.0.smithi079.stderr:2024-04-23T16:39:38.694+0000 7f4cfafff7c0 -1 osd.0 0 log_to_monitors true 2024-04-23T16:39:38.700 INFO:tasks.ceph.osd.1.smithi079.stderr:2024-04-23T16:39:38.698+0000 7f5dc87fa7c0 -1 osd.1 0 log_to_monitors true 2024-04-23T16:39:39.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-04-23T16:39:39.680+0000 7f9ad86bd640 -1 osd.5 0 waiting for initial osdmap 2024-04-23T16:39:39.683 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-04-23T16:39:39.680+0000 7f52146d2640 -1 osd.3 0 waiting for initial osdmap 2024-04-23T16:39:39.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-04-23T16:39:39.684+0000 7f9ad44f9640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:39:39.692 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-04-23T16:39:39.688+0000 7f521050e640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:39:40.685 INFO:tasks.ceph.osd.0.smithi079.stderr:2024-04-23T16:39:40.682+0000 7f4cf6fa8640 -1 osd.0 0 waiting for initial osdmap 2024-04-23T16:39:40.686 INFO:tasks.ceph.osd.2.smithi079.stderr:2024-04-23T16:39:40.682+0000 7fa595419640 -1 osd.2 0 waiting for initial osdmap 2024-04-23T16:39:40.693 INFO:tasks.ceph.osd.2.smithi079.stderr:2024-04-23T16:39:40.690+0000 7fa591255640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:39:40.695 INFO:tasks.ceph.osd.0.smithi079.stderr:2024-04-23T16:39:40.690+0000 7f4cf2de4640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:39:41.340 INFO:tasks.ceph.osd.1.smithi079.stderr:2024-04-23T16:39:41.338+0000 7f5dc47a3640 -1 osd.1 0 waiting for initial osdmap 2024-04-23T16:39:41.344 INFO:tasks.ceph.osd.1.smithi079.stderr:2024-04-23T16:39:41.342+0000 7f5dbc5d7640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:39:41.348 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-04-23T16:39:41.344+0000 7f4ce38e4640 -1 osd.4 0 waiting for initial osdmap 2024-04-23T16:39:41.356 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-04-23T16:39:41.352+0000 7f4cdb718640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:39:41.802 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T16:39:41.922 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.918+0000 7fcea5912640 10 monclient: get_monmap_and_config 2024-04-23T16:39:41.922 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.918+0000 7fcea5912640 10 monclient: build_initial_monmap 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.918+0000 7fcea5912640 10 monclient: monmap: 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:39:41.924279+0000 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:39:41.924279+0000 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:41.923 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.918+0000 7fcea5912640 1 Processor -- start 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.918+0000 7fcea5912640 1 -- start start 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.918+0000 7fcea5912640 10 monclient: init 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcea00a1430 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcea009dc20 0x7fcea01396b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): picked mon.noname-a con 0x7fcea009dc20 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:41.926 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcea00a1f20 0x7fcea013bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): picked mon.noname-c con 0x7fcea00a1f20 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea009e6e0 con 0x7fcea00a1430 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fcea009d4e0 con 0x7fcea009dc20 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcea009d9c0 con 0x7fcea00a1f20 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fcea5912640 10 monclient(hunting): authenticate will time out at 936.112160s 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcea009dc20 0x7fcea01396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcea009dc20 0x7fcea01396b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47666/0 (socket says 172.21.15.79:47666) 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 -- 172.21.15.79:0/4052133585 learned_addr learned my addr 172.21.15.79:0/4052133585 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcea009dc20 auth_method 0 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:41.927 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9f7fe640 1 --2- 172.21.15.79:0/4052133585 >> v2:172.21.15.204:3300/0 conn(0x7fcea00a1f20 0x7fcea013bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcea00a1f20 auth_method 0 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 -- 172.21.15.79:0/4052133585 >> v2:172.21.15.204:3300/0 conn(0x7fcea00a1f20 msgr2=0x7fcea013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 --2- 172.21.15.79:0/4052133585 >> v2:172.21.15.204:3300/0 conn(0x7fcea00a1f20 0x7fcea013bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 -- 172.21.15.79:0/4052133585 >> v1:172.21.15.204:6789/0 conn(0x7fcea00a1430 legacy=0x7fcea00a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient: _finish_hunting 0 2024-04-23T16:39:41.928 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 monclient: found mon.noname-a 2024-04-23T16:39:41.929 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:41.929 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:41.929 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.922+0000 7fce9e7fc640 1 -- 172.21.15.79:0/4052133585 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea009e9c0 con 0x7fcea009dc20 2024-04-23T16:39:41.932 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 -- 172.21.15.79:0/4052133585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 msgr2=0x7fcea01396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 --2- 172.21.15.79:0/4052133585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01396b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce90009870 tx=0x7fce9001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c665a4a5dca5a455 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 1 -- 172.21.15.79:0/4052133585 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce900b0070 con 0x7fcea009dc20 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: dump: 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:41.933 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:41.934 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 1 -- 172.21.15.79:0/4052133585 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce900ab600 con 0x7fcea009dc20 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 1 -- 172.21.15.79:0/4052133585 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce900ab9d0 con 0x7fcea009dc20 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9dffb640 10 monclient: dump: 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:41.935 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 5 monclient: authenticate success, global_id 4179 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 4 set_mon_vals no callback set 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: shutdown 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- 172.21.15.79:0/4052133585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 msgr2=0x7fcea01396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 --2- 172.21.15.79:0/4052133585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01396b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce90009870 tx=0x7fce9001cc80 comp rx=0 tx=0).stop 2024-04-23T16:39:41.936 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- 172.21.15.79:0/4052133585 shutdown_connections 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 --2- 172.21.15.79:0/4052133585 >> v2:172.21.15.204:3300/0 conn(0x7fcea00a1f20 0x7fcea013bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 --2- 172.21.15.79:0/4052133585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01396b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- 172.21.15.79:0/4052133585 >> 172.21.15.79:0/4052133585 conn(0x7fcea00a6330 msgr2=0x7fcea00a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- 172.21.15.79:0/4052133585 shutdown_connections 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- 172.21.15.79:0/4052133585 wait complete. 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: build_initial_monmap 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: monmap: 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-04-23T16:39:41.937 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 Processor -- start 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- start start 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: init 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:41.938 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): picked mon.noname-a con 0x7fcea009dc20 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 0x7fcea01cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): picked mon.noname-c con 0x7fcea00a1430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcea00a1f20 0x7fcea01375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): picked mon.noname-b con 0x7fcea00a1f20 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcea013eae0 con 0x7fcea009dc20 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcea013e960 con 0x7fcea00a1f20 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcea013ec60 con 0x7fcea00a1430 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:41.939 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fcea5912640 10 monclient(hunting): authenticate will time out at 936.119398s 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 0x7fcea01cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 0x7fcea01cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47034/0 (socket says 172.21.15.79:47034) 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 -- 172.21.15.79:0/2156022469 learned_addr learned my addr 172.21.15.79:0/2156022469 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcea00a1430 auth_method 0 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): get_auth_request con 0x7fcea009dc20 auth_method 0 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:41.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9f7fe640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcea00a1f20 0x7fcea01375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcea00a1f20 auth_method 0 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2024-04-23T16:39:41.941 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 -- 172.21.15.79:0/2156022469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcea00a1f20 msgr2=0x7fcea01375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcea00a1f20 0x7fcea01375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 -- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 msgr2=0x7fcea01cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.926+0000 7fce9e7fc640 10 monclient: _finish_hunting 0 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9e7fc640 1 monclient: found mon.noname-c 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9e7fc640 1 -- 172.21.15.79:0/2156022469 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea0139180 con 0x7fcea00a1430 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9effd640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:41.942 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9f7fe640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcea00a1f20 0x7fcea01375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9e7fc640 1 -- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 msgr2=0x7fcea01cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9e7fc640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 0x7fcea01cd840 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fce900ae040 tx=0x7fce9001aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce900ba020 con 0x7fcea00a1430 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: dump: 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.945 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 5 monclient: authenticate success, global_id 4160 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce900b6630 con 0x7fcea00a1430 2024-04-23T16:39:41.946 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce900b0040 con 0x7fcea00a1430 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 monclient: _renew_subs 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea009e070 con 0x7fcea00a1430 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce9d7fa640 4 set_mon_vals no callback set 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 10 monclient: dump: 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.947 INFO:teuthology.misc.health.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr: 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 monclient: _renew_subs 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea013abd0 con 0x7fcea00a1430 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:41.948 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea004fcf0 con 0x7fcea00a1430 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7fce900bd020 con 0x7fcea00a1430 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 1 --2- 172.21.15.79:0/2156022469 >> v2:172.21.15.204:6812/722020384 conn(0x7fce80036350 0x7fce80038810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.930+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(12..12 src has 1..12) ==== 3142+0+0 (secure 0 0 0) 0x7fce900e2f40 con 0x7fcea00a1430 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.934+0000 7fce9effd640 1 --2- 172.21.15.79:0/2156022469 >> v2:172.21.15.204:6812/722020384 conn(0x7fce80036350 0x7fce80038810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.934+0000 7fce9effd640 10 monclient: get_auth_request con 0x7fce80036350 auth_method 0 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.934+0000 7fce9effd640 1 --2- 172.21.15.79:0/2156022469 >> v2:172.21.15.204:6812/722020384 conn(0x7fce80036350 0x7fce80038810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcea013a970 tx=0x7fce88007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.938+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce900c4030 con 0x7fcea00a1430 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.938+0000 7fce97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:41.949 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:41.938+0000 7fce97fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:42.101 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fcea5912640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:42.101 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fcea5912640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:42.101 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fcea5912640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:42.102 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcea006b9c0 con 0x7fcea00a1430 2024-04-23T16:39:42.103 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fce97fff640 1 -- 172.21.15.79:0/2156022469 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+8937 (secure 0 0 0) 0x7fcea009e070 con 0x7fcea00a1430 2024-04-23T16:39:42.103 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fce97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:42.103 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.098+0000 7fce97fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:42.103 INFO:teuthology.misc.health.smithi079.stdout: 2024-04-23T16:39:42.103 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":12,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:39:41.690496+0000","last_up_change":"2024-04-23T16:39:41.690496+0000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:39:41.354561+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":987647954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":987647954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":987647954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":987647954}]},"public_addr":"172.21.15.79:6802/987647954","cluster_addr":"172.21.15.79:6805/987647954","heartbeat_back_addr":"172.21.15.79:6810/987647954","heartbeat_front_addr":"172.21.15.79:6808/987647954","state":["exists","up"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3615156451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":3615156451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":3615156451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":3615156451}]},"public_addr":"172.21.15.79:6803/3615156451","cluster_addr":"172.21.15.79:6806/3615156451","heartbeat_back_addr":"172.21.15.79:6811/3615156451","heartbeat_front_addr":"172.21.15.79:6809/3615156451","state":["exists","up"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3678369310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":3678369310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":3678369310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":3678369310}]},"public_addr":"172.21.15.79:6800/3678369310","cluster_addr":"172.21.15.79:6801/3678369310","heartbeat_back_addr":"172.21.15.79:6807/3678369310","heartbeat_front_addr":"172.21.15.79:6804/3678369310","state":["exists","up"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3194378740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":3194378740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":3194378740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3194378740}]},"public_addr":"172.21.15.204:6800/3194378740","cluster_addr":"172.21.15.204:6801/3194378740","heartbeat_back_addr":"172.21.15.204:6803/3194378740","heartbeat_front_addr":"172.21.15.204:6802/3194378740","state":["exists","up"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3419064638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":3419064638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":3419064638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3419064638}]},"public_addr":"172.21.15.204:6804/3419064638","cluster_addr":"172.21.15.204:6805/3419064638","heartbeat_back_addr":"172.21.15.204:6807/3419064638","heartbeat_front_addr":"172.21.15.204:6806/3419064638","state":["exists","up"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":3961528928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":3961528928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":3961528928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3961528928}]},"public_addr":"172.21.15.204:6808/3961528928","cluster_addr":"172.21.15.204:6809/3961528928","heartbeat_back_addr":"172.21.15.204:6811/3961528928","heartbeat_front_addr":"172.21.15.204:6810/3961528928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.480934+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.584524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:39:42.105 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 >> v2:172.21.15.204:6812/722020384 conn(0x7fce80036350 msgr2=0x7fce80038810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.105 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 --2- 172.21.15.79:0/2156022469 >> v2:172.21.15.204:6812/722020384 conn(0x7fce80036350 0x7fce80038810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcea013a970 tx=0x7fce88007e40 comp rx=0 tx=0).stop 2024-04-23T16:39:42.105 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 10 monclient: shutdown 2024-04-23T16:39:42.105 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:42.105 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 msgr2=0x7fcea01cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.105 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 0x7fcea01cd840 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fce900ae040 tx=0x7fce9001aa60 comp rx=0 tx=0).stop 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 shutdown_connections 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 --2- 172.21.15.79:0/2156022469 >> v2:172.21.15.204:6812/722020384 conn(0x7fce80036350 0x7fce80038810 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcea00a1f20 0x7fcea01375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcea00a1430 0x7fcea01cd840 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 --2- 172.21.15.79:0/2156022469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcea009dc20 0x7fcea01cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 >> 172.21.15.79:0/2156022469 conn(0x7fcea00a6330 msgr2=0x7fcea00726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 shutdown_connections 2024-04-23T16:39:42.106 INFO:teuthology.misc.health.smithi079.stderr:2024-04-23T16:39:42.102+0000 7fcea5912640 1 -- 172.21.15.79:0/2156022469 wait complete. 2024-04-23T16:39:42.121 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-23T16:39:42.121 INFO:tasks.ceph:Creating RBD pool 2024-04-23T16:39:42.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-23T16:39:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.238+0000 7f6a6a150640 10 monclient: get_monmap_and_config 2024-04-23T16:39:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.238+0000 7f6a6a150640 10 monclient: build_initial_monmap 2024-04-23T16:39:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.238+0000 7f6a6a150640 10 monclient: monmap: 2024-04-23T16:39:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:42.245070+0000 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:42.245070+0000 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:42.244 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 Processor -- start 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 -- start start 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient: init 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6a641355c0 0x7f6a641359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): picked mon.noname-c con 0x7f6a641355c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f6a64135ee0 0x7f6a640a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): picked mon.noname-a con 0x7f6a64135ee0 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a6409c690 0x7f6a640a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): picked mon.noname-b con 0x7f6a6409c690 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:42.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f6a64072bb0 con 0x7f6a64135ee0 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f6a6409bdc0 con 0x7f6a641355c0 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6409f6c0 con 0x7f6a6409c690 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a6a150640 10 monclient(hunting): authenticate will time out at 936.432879s 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f6a64135ee0 0x7f6a640a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a6409c690 0x7f6a640a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f6a64135ee0 0x7f6a640a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47688/0 (socket says 172.21.15.79:47688) 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a6409c690 0x7f6a640a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47044/0 (socket says 172.21.15.79:47044) 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 -- 172.21.15.79:0/3542448082 learned_addr learned my addr 172.21.15.79:0/3542448082 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): get_auth_request con 0x7f6a64135ee0 auth_method 0 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): get_auth_request con 0x7f6a6409c690 auth_method 0 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a637fe640 1 --2- 172.21.15.79:0/3542448082 >> v2:172.21.15.204:3300/0 conn(0x7f6a641355c0 0x7f6a641359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_done global_id 4185 payload 274 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 -- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a6409c690 msgr2=0x7f6a640a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 --2- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a6409c690 0x7f6a640a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 -- 172.21.15.79:0/3542448082 >> v2:172.21.15.204:3300/0 conn(0x7f6a641355c0 msgr2=0x7f6a641359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 --2- 172.21.15.79:0/3542448082 >> v2:172.21.15.204:3300/0 conn(0x7f6a641355c0 0x7f6a641359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 monclient: found mon.noname-a 2024-04-23T16:39:42.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:42.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a5bfff640 1 -- 172.21.15.79:0/3542448082 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6409f1e0 con 0x7f6a64135ee0 2024-04-23T16:39:42.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a637fe640 10 monclient: get_auth_request con 0x7f6a641355c0 auth_method 0 2024-04-23T16:39:42.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.242+0000 7f6a637fe640 1 --2- 172.21.15.79:0/3542448082 >> v2:172.21.15.204:3300/0 conn(0x7f6a641355c0 0x7f6a641359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a5bfff640 1 -- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 msgr2=0x7f6a640a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a5bfff640 1 --2- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a640a13e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a6409c1d0 tx=0x7f6a4c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc59dd23ae26caec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 1 -- 172.21.15.79:0/3542448082 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0b0070 con 0x7f6a64135ee0 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: dump: 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.255 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 1 -- 172.21.15.79:0/3542448082 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0ab600 con 0x7f6a64135ee0 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 1 -- 172.21.15.79:0/3542448082 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0ab9d0 con 0x7f6a64135ee0 2024-04-23T16:39:42.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 5 monclient: authenticate success, global_id 4185 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 4 set_mon_vals no callback set 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a62ffd640 10 monclient: dump: 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.257 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 10 monclient: shutdown 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 -- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 msgr2=0x7f6a640a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 --2- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a640a13e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a6409c1d0 tx=0x7f6a4c01c3e0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 -- 172.21.15.79:0/3542448082 shutdown_connections 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 --2- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a6409c690 0x7f6a640a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 --2- 172.21.15.79:0/3542448082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a640a13e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 --2- 172.21.15.79:0/3542448082 >> v2:172.21.15.204:3300/0 conn(0x7f6a641355c0 0x7f6a641359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 -- 172.21.15.79:0/3542448082 >> 172.21.15.79:0/3542448082 conn(0x7f6a640a6250 msgr2=0x7f6a640a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 -- 172.21.15.79:0/3542448082 shutdown_connections 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.246+0000 7f6a6a150640 1 -- 172.21.15.79:0/3542448082 wait complete. 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient: build_initial_monmap 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient: monmap: 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.259 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 Processor -- start 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 -- start start 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient: init 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6409c690 0x7f6a641cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): picked mon.noname-b con 0x7f6a6409c690 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 0x7f6a64134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): picked mon.noname-c con 0x7f6a641355c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a641ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): picked mon.noname-a con 0x7f6a64135ee0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6413ef80 con 0x7f6a64135ee0 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6413f100 con 0x7f6a6409c690 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6413f280 con 0x7f6a641355c0 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:42.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a6a150640 10 monclient(hunting): authenticate will time out at 936.441128s 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a641ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 0x7f6a64134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 0x7f6a64134650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47048/0 (socket says 172.21.15.79:47048) 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 1 -- 172.21.15.79:0/626042215 learned_addr learned my addr 172.21.15.79:0/626042215 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a641ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47698/0 (socket says 172.21.15.79:47698) 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): get_auth_request con 0x7f6a641355c0 auth_method 0 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): get_auth_request con 0x7f6a64135ee0 auth_method 0 2024-04-23T16:39:42.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a637fe640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6409c690 0x7f6a641cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a637fe640 10 monclient(hunting): get_auth_request con 0x7f6a6409c690 auth_method 0 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a637fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 -- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 msgr2=0x7f6a64134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 0x7f6a64134650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 -- 172.21.15.79:0/626042215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6409c690 msgr2=0x7f6a641cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6409c690 0x7f6a641cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 monclient: found mon.noname-a 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 -- 172.21.15.79:0/626042215 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6411dd00 con 0x7f6a64135ee0 2024-04-23T16:39:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a5bfff640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 0x7f6a64134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:42.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 -- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 msgr2=0x7f6a641ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.250+0000 7f6a63fff640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a641ca500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00ef90 tx=0x7f6a5c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c018020 con 0x7f6a64135ee0 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: dump: 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.267 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: _finish_auth 0 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0146a0 con 0x7f6a64135ee0 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 5 monclient: authenticate success, global_id 4188 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 monclient: _renew_subs 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6411d520 con 0x7f6a64135ee0 2024-04-23T16:39:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c014a90 con 0x7f6a64135ee0 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 10 monclient: dump: 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.269 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 monclient: _renew_subs 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a6411eba0 con 0x7f6a64135ee0 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a627fc640 4 set_mon_vals no callback set 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f6a5c01b020 con 0x7f6a64135ee0 2024-04-23T16:39:42.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a640692e0 con 0x7f6a64135ee0 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 1 --2- 172.21.15.79:0/626042215 >> v2:172.21.15.204:6812/722020384 conn(0x7f6a440363a0 0x7f6a44038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3142+0+0 (secure 0 0 0) 0x7f6a5c049a70 con 0x7f6a64135ee0 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a637fe640 1 --2- 172.21.15.79:0/626042215 >> v2:172.21.15.204:6812/722020384 conn(0x7f6a440363a0 0x7f6a44038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.254+0000 7f6a637fe640 10 monclient: get_auth_request con 0x7f6a440363a0 auth_method 0 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.258+0000 7f6a637fe640 1 --2- 172.21.15.79:0/626042215 >> v2:172.21.15.204:6812/722020384 conn(0x7f6a440363a0 0x7f6a44038860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a54006fd0 tx=0x7f6a54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.258+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6a5c022030 con 0x7f6a64135ee0 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.258+0000 7f6a60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.258+0000 7f6a60ff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.426+0000 7f6a6a150640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-23T16:39:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.426+0000 7f6a6a150640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-23T16:39:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.426+0000 7f6a6a150640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.426+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f6a64071bf0 con 0x7f6a64135ee0 2024-04-23T16:39:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.694+0000 7f6a60ff9640 1 -- 172.21.15.79:0/626042215 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7f6a6411d520 con 0x7f6a64135ee0 2024-04-23T16:39:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.694+0000 7f6a60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-23T16:39:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.694+0000 7f6a60ff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-23T16:39:42.697 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'rbd' created 2024-04-23T16:39:42.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 >> v2:172.21.15.204:6812/722020384 conn(0x7f6a440363a0 msgr2=0x7f6a44038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 --2- 172.21.15.79:0/626042215 >> v2:172.21.15.204:6812/722020384 conn(0x7f6a440363a0 0x7f6a44038860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a54006fd0 tx=0x7f6a54008040 comp rx=0 tx=0).stop 2024-04-23T16:39:42.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 10 monclient: shutdown 2024-04-23T16:39:42.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:42.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 msgr2=0x7f6a641ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a641ca500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00ef90 tx=0x7f6a5c00c490 comp rx=0 tx=0).stop 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 shutdown_connections 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 --2- 172.21.15.79:0/626042215 >> v2:172.21.15.204:6812/722020384 conn(0x7f6a440363a0 0x7f6a44038860 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a64135ee0 0x7f6a641ca500 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f6a641355c0 0x7f6a64134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 --2- 172.21.15.79:0/626042215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6409c690 0x7f6a641cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 >> 172.21.15.79:0/626042215 conn(0x7f6a640a6250 msgr2=0x7f6a640a3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 shutdown_connections 2024-04-23T16:39:42.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.698+0000 7f6a6a150640 1 -- 172.21.15.79:0/626042215 wait complete. 2024-04-23T16:39:42.726 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient: get_monmap_and_config 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient: build_initial_monmap 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient: monmap: 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:42.854763+0000 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:42.854763+0000 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:42.854 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 1 Processor -- start 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 1 -- start start 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient: init 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 0x7f575c1349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient(hunting): picked mon.noname-b con 0x7f575c1345d0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f575c134ef0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f575c09f170 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c0a22e0 con 0x7f575c134ef0 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c09c350 con 0x7f575c09f170 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f575c09c950 con 0x7f575c1345d0 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f57628cb640 10 monclient(hunting): authenticate will time out at 937.042580s 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 0x7f575c1349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 0x7f575c1349b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47064/0 (socket says 172.21.15.79:47064) 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 -- 172.21.15.79:0/1134566965 learned_addr learned my addr 172.21.15.79:0/1134566965 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): get_auth_request con 0x7f575c1345d0 auth_method 0 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575b7fe640 1 --1- 172.21.15.79:0/1134566965 >> v1:172.21.15.79:6789/0 conn(0x7f575c134ef0 0x7f575c0a14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:51202/0 (socket says 172.21.15.79:51202) 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 -- 172.21.15.79:0/1134566965 >> v1:172.21.15.204:6789/0 conn(0x7f575c09f170 legacy=0x7f575c0a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 -- 172.21.15.79:0/1134566965 >> v1:172.21.15.79:6789/0 conn(0x7f575c134ef0 legacy=0x7f575c0a14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 monclient: found mon.noname-b 2024-04-23T16:39:42.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:42.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:42.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.849+0000 7f575bfff640 1 -- 172.21.15.79:0/1134566965 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c09f810 con 0x7f575c1345d0 2024-04-23T16:39:42.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575bfff640 1 -- 172.21.15.79:0/1134566965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 msgr2=0x7f575c1349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:42.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575bfff640 1 --2- 172.21.15.79:0/1134566965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 0x7f575c1349b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f574c0098d0 tx=0x7f574c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9624c292b633dd0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:42.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 1 -- 172.21.15.79:0/1134566965 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c0b0070 con 0x7f575c1345d0 2024-04-23T16:39:42.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:42.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: dump: 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: _finish_auth 0 2024-04-23T16:39:42.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 1 -- 172.21.15.79:0/1134566965 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f574c0ab600 con 0x7f575c1345d0 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 1 -- 172.21.15.79:0/1134566965 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c0ab9d0 con 0x7f575c1345d0 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f57628cb640 5 monclient: authenticate success, global_id 4169 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f57628cb640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f57628cb640 4 set_mon_vals no callback set 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.853+0000 7f575affd640 10 monclient: dump: 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.862 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient: shutdown 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- 172.21.15.79:0/1134566965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 msgr2=0x7f575c1349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 --2- 172.21.15.79:0/1134566965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 0x7f575c1349b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f574c0098d0 tx=0x7f574c01c3c0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- 172.21.15.79:0/1134566965 shutdown_connections 2024-04-23T16:39:42.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 --2- 172.21.15.79:0/1134566965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c1345d0 0x7f575c1349b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- 172.21.15.79:0/1134566965 >> 172.21.15.79:0/1134566965 conn(0x7f575c0a25b0 msgr2=0x7f575c0a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:42.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- 172.21.15.79:0/1134566965 shutdown_connections 2024-04-23T16:39:42.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- 172.21.15.79:0/1134566965 wait complete. 2024-04-23T16:39:42.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient: build_initial_monmap 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient: monmap: 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 Processor -- start 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- start start 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient: init 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 0x7f575c0713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): picked mon.noname-a con 0x7f575c09f170 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f575c1345d0 0x7f575c133920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): picked mon.noname-b con 0x7f575c1345d0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c134ef0 0x7f575c11c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): picked mon.noname-c con 0x7f575c134ef0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f575c141060 con 0x7f575c09f170 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f575c140ee0 con 0x7f575c1345d0 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f575c1411e0 con 0x7f575c134ef0 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f57628cb640 10 monclient(hunting): authenticate will time out at 937.050218s 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 0x7f575c0713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 0x7f575c0713c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47704/0 (socket says 172.21.15.79:47704) 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 -- 172.21.15.79:0/4024062282 learned_addr learned my addr 172.21.15.79:0/4024062282 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): get_auth_request con 0x7f575c09f170 auth_method 0 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c134ef0 0x7f575c11c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): get_auth_request con 0x7f575c134ef0 auth_method 0 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f5760e68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 -- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c134ef0 msgr2=0x7f575c11c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c134ef0 0x7f575c11c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 -- 172.21.15.79:0/4024062282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f575c1345d0 msgr2=0x7f575c133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575b7fe640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f575c1345d0 0x7f575c133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f575c1345d0 0x7f575c133920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 monclient: found mon.noname-a 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.857+0000 7f575bfff640 1 -- 172.21.15.79:0/4024062282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c11d8b0 con 0x7f575c09f170 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f575bfff640 1 -- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 msgr2=0x7f575c0713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f575bfff640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 0x7f575c0713c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f574c01cc60 tx=0x7f574c019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c0b0050 con 0x7f575c09f170 2024-04-23T16:39:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: dump: 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.874 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: _finish_auth 0 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f574c0b1040 con 0x7f575c09f170 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 5 monclient: authenticate success, global_id 4191 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 monclient: _renew_subs 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575c072a50 con 0x7f575c09f170 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c0abe30 con 0x7f575c09f170 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 10 monclient: dump: 2024-04-23T16:39:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f575a7fc640 4 set_mon_vals no callback set 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 monclient: _renew_subs 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f575c11f2c0 con 0x7f575c09f170 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f574c0bc020 con 0x7f575c09f170 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f575c0692e0 con 0x7f575c09f170 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 1 --2- 172.21.15.79:0/4024062282 >> v2:172.21.15.204:6812/722020384 conn(0x7f57480363a0 0x7f5748038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3495+0+0 (secure 0 0 0) 0x7f574c0e26c0 con 0x7f575c09f170 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f575b7fe640 1 --2- 172.21.15.79:0/4024062282 >> v2:172.21.15.204:6812/722020384 conn(0x7f57480363a0 0x7f5748038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:42.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.861+0000 7f575b7fe640 10 monclient: get_auth_request con 0x7f57480363a0 auth_method 0 2024-04-23T16:39:42.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.865+0000 7f575b7fe640 1 --2- 172.21.15.79:0/4024062282 >> v2:172.21.15.204:6812/722020384 conn(0x7f57480363a0 0x7f5748038860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f575c11e5f0 tx=0x7f5750009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:42.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.865+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f574c0a4e20 con 0x7f575c09f170 2024-04-23T16:39:42.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.865+0000 7f5758ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:42.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:42.865+0000 7f5758ff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:43.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.029+0000 7f57628cb640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-23T16:39:43.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.029+0000 7f57628cb640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-23T16:39:43.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.029+0000 7f57628cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:43.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.029+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f575c0724a0 con 0x7f575c09f170 2024-04-23T16:39:43.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.693+0000 7f5758ff9640 1 -- 172.21.15.79:0/4024062282 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 173+0+0 (secure 0 0 0) 0x7f574c0160d0 con 0x7f575c09f170 2024-04-23T16:39:43.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.693+0000 7f5758ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-23T16:39:43.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.693+0000 7f5758ff9640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-23T16:39:43.699 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 >> v2:172.21.15.204:6812/722020384 conn(0x7f57480363a0 msgr2=0x7f5748038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 --2- 172.21.15.79:0/4024062282 >> v2:172.21.15.204:6812/722020384 conn(0x7f57480363a0 0x7f5748038860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f575c11e5f0 tx=0x7f5750009040 comp rx=0 tx=0).stop 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 10 monclient: shutdown 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 msgr2=0x7f575c0713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 0x7f575c0713c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f574c01cc60 tx=0x7f574c019160 comp rx=0 tx=0).stop 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 shutdown_connections 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 --2- 172.21.15.79:0/4024062282 >> v2:172.21.15.204:6812/722020384 conn(0x7f57480363a0 0x7f5748038860 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f575c134ef0 0x7f575c11c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f575c1345d0 0x7f575c133920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 --2- 172.21.15.79:0/4024062282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f575c09f170 0x7f575c0713c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 >> 172.21.15.79:0/4024062282 conn(0x7f575c0a25b0 msgr2=0x7f575c0a38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:43.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 shutdown_connections 2024-04-23T16:39:43.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.697+0000 7f57628cb640 1 -- 172.21.15.79:0/4024062282 wait complete. 2024-04-23T16:39:43.732 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-23T16:39:43.733 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T16:39:43.733 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-23T16:39:43.733 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-23T16:39:43.733 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-23T16:39:43.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.841+0000 7f483ad2e640 10 monclient: get_monmap_and_config 2024-04-23T16:39:43.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.841+0000 7f483ad2e640 10 monclient: build_initial_monmap 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient: monmap: 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:43.849625+0000 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:43.849625+0000 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:43.848 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:43.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 Processor -- start 2024-04-23T16:39:43.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 -- start start 2024-04-23T16:39:43.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient: init 2024-04-23T16:39:43.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:43.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f483406ca30 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48340a4c80 0x7f483413c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): picked mon.noname-c con 0x7f48340a4c80 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 0x7f483413f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): picked mon.noname-b con 0x7f483413ce00 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48340a17b0 con 0x7f483406ca30 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f48340a50b0 con 0x7f48340a4c80 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f483406ce10 con 0x7f483413ce00 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f483ad2e640 10 monclient(hunting): authenticate will time out at 938.037418s 2024-04-23T16:39:43.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 0x7f483413f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 0x7f483413f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47076/0 (socket says 172.21.15.79:47076) 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 -- 172.21.15.79:0/3888687193 learned_addr learned my addr 172.21.15.79:0/3888687193 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): get_auth_request con 0x7f483413ce00 auth_method 0 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f4833fff640 1 --2- 172.21.15.79:0/3888687193 >> v2:172.21.15.204:3300/0 conn(0x7f48340a4c80 0x7f483413c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:43.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f4833fff640 10 monclient(hunting): get_auth_request con 0x7f48340a4c80 auth_method 0 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f4833fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f4833fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f4833fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 -- 172.21.15.79:0/3888687193 >> v2:172.21.15.204:3300/0 conn(0x7f48340a4c80 msgr2=0x7f483413c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 --2- 172.21.15.79:0/3888687193 >> v2:172.21.15.204:3300/0 conn(0x7f48340a4c80 0x7f483413c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 -- 172.21.15.79:0/3888687193 >> v1:172.21.15.204:6789/0 conn(0x7f483406ca30 legacy=0x7f48340a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient: _finish_hunting 0 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 monclient: found mon.noname-b 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:43.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.845+0000 7f48392cb640 1 -- 172.21.15.79:0/3888687193 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48340a14d0 con 0x7f483413ce00 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48392cb640 1 -- 172.21.15.79:0/3888687193 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 msgr2=0x7f483413f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48392cb640 1 --2- 172.21.15.79:0/3888687193 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 0x7f483413f1f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f482c009e50 tx=0x7f482c01c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e7dc7afcd8b7bdd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 1 -- 172.21.15.79:0/3888687193 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c0ac020 con 0x7f483413ce00 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: dump: 2024-04-23T16:39:43.859 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 1 -- 172.21.15.79:0/3888687193 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482c0a6450 con 0x7f483413ce00 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 1 -- 172.21.15.79:0/3888687193 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c0a6820 con 0x7f483413ce00 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 5 monclient: authenticate success, global_id 4181 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 4 set_mon_vals no callback set 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f48337fe640 10 monclient: dump: 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 10 monclient: shutdown 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 -- 172.21.15.79:0/3888687193 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 msgr2=0x7f483413f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 --2- 172.21.15.79:0/3888687193 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 0x7f483413f1f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f482c009e50 tx=0x7f482c01c6d0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 -- 172.21.15.79:0/3888687193 shutdown_connections 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 --2- 172.21.15.79:0/3888687193 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f483413ce00 0x7f483413f1f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 --2- 172.21.15.79:0/3888687193 >> v2:172.21.15.204:3300/0 conn(0x7f48340a4c80 0x7f483413c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 -- 172.21.15.79:0/3888687193 >> 172.21.15.79:0/3888687193 conn(0x7f48340aa370 msgr2=0x7f48340a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 -- 172.21.15.79:0/3888687193 shutdown_connections 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 -- 172.21.15.79:0/3888687193 wait complete. 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 10 monclient: build_initial_monmap 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 10 monclient: monmap: 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:43.863 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 Processor -- start 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 1 -- start start 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.849+0000 7f483ad2e640 10 monclient: init 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f483406ca30 0x7f48341cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): picked mon.noname-b con 0x7f483406ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 0x7f4834135510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:43.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): picked mon.noname-c con 0x7f48340a4c80 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f483413ce00 0x7f483409c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): picked mon.noname-a con 0x7f483413ce00 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48341420d0 con 0x7f483413ce00 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4834141f50 con 0x7f483406ca30 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4834142250 con 0x7f48340a4c80 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient(hunting): authenticate will time out at 938.044390s 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 0x7f4834135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f483413ce00 0x7f483409c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:43.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 0x7f4834135510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47084/0 (socket says 172.21.15.79:47084) 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 -- 172.21.15.79:0/4009788122 learned_addr learned my addr 172.21.15.79:0/4009788122 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): get_auth_request con 0x7f48340a4c80 auth_method 0 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): get_auth_request con 0x7f483413ce00 auth_method 0 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:43.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48392cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4838aca640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f483406ca30 0x7f48341cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4838aca640 10 monclient(hunting): get_auth_request con 0x7f483406ca30 auth_method 0 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4838aca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4838aca640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4838aca640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 -- 172.21.15.79:0/4009788122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f483406ca30 msgr2=0x7f48341cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f483406ca30 0x7f48341cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 -- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f483413ce00 msgr2=0x7f483409c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f483413ce00 0x7f483409c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:43.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 monclient: found mon.noname-c 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 -- 172.21.15.79:0/4009788122 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483409e370 con 0x7f48340a4c80 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 -- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 msgr2=0x7f4834135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4833fff640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 0x7f4834135510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f482800d800 tx=0x7f482800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482800b8d0 con 0x7f48340a4c80 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: dump: 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:43.868 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:43.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4828019070 con 0x7f48340a4c80 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 5 monclient: authenticate success, global_id 4184 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient: _renew_subs 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48340a9280 con 0x7f48340a4c80 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482800bcf0 con 0x7f48340a4c80 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f48317fa640 10 monclient: dump: 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:43.870 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient: _renew_subs 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48341d0150 con 0x7f48340a4c80 2024-04-23T16:39:43.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.853+0000 7f4832ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:43.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f483ad2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:43.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f482801e460 con 0x7f48340a4c80 2024-04-23T16:39:43.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f483ad2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f483ad2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48340a4060 con 0x7f48340a4c80 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f48317fa640 1 --2- 172.21.15.79:0/4009788122 >> v2:172.21.15.204:6812/722020384 conn(0x7f4820036350 0x7f4820038810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (secure 0 0 0) 0x7f48280493b0 con 0x7f48340a4c80 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f4838aca640 1 --2- 172.21.15.79:0/4009788122 >> v2:172.21.15.204:6812/722020384 conn(0x7f4820036350 0x7f4820038810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f4838aca640 10 monclient: get_auth_request con 0x7f4820036350 auth_method 0 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.857+0000 7f4838aca640 1 --2- 172.21.15.79:0/4009788122 >> v2:172.21.15.204:6812/722020384 conn(0x7f4820036350 0x7f4820038810 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4824004e30 tx=0x7f4824009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.861+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f48340a4060 con 0x7f48340a4c80 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.861+0000 7f48317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:43.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:43.861+0000 7f48317fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:44.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.065+0000 7f483ad2e640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-23T16:39:44.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.065+0000 7f483ad2e640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-23T16:39:44.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.065+0000 7f483ad2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.065+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f48340692e0 con 0x7f48340a4c80 2024-04-23T16:39:44.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.069+0000 7f48317fa640 1 -- 172.21.15.79:0/4009788122 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+84622 (secure 0 0 0) 0x7f48280482b0 con 0x7f48340a4c80 2024-04-23T16:39:44.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.069+0000 7f48317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-23T16:39:44.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.069+0000 7f48317fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:44.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 >> v2:172.21.15.204:6812/722020384 conn(0x7f4820036350 msgr2=0x7f4820038810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 --2- 172.21.15.79:0/4009788122 >> v2:172.21.15.204:6812/722020384 conn(0x7f4820036350 0x7f4820038810 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4824004e30 tx=0x7f4824009040 comp rx=0 tx=0).stop 2024-04-23T16:39:44.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 10 monclient: shutdown 2024-04-23T16:39:44.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 msgr2=0x7f4834135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 0x7f4834135510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f482800d800 tx=0x7f482800dcc0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 shutdown_connections 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 --2- 172.21.15.79:0/4009788122 >> v2:172.21.15.204:6812/722020384 conn(0x7f4820036350 0x7f4820038810 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f483413ce00 0x7f483409c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f48340a4c80 0x7f4834135510 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 --2- 172.21.15.79:0/4009788122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f483406ca30 0x7f48341cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 >> 172.21.15.79:0/4009788122 conn(0x7f48340aa370 msgr2=0x7f48340a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 shutdown_connections 2024-04-23T16:39:44.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.073+0000 7f483ad2e640 1 -- 172.21.15.79:0/4009788122 wait complete. 2024-04-23T16:39:44.094 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":722020384}]},"active_addr":"172.21.15.204:6812/722020384","active_change":"2024-04-23T16:39:35.284716+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":3380088065}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":4073722920}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":765745400}]}]} 2024-04-23T16:39:44.096 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-23T16:39:44.096 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-23T16:39:44.096 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T16:39:44.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient: monmap: 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.215625+0000 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.215625+0000 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.215 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 Processor -- start 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 -- start start 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient: init 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fe84806ca30 0x7fe8480a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): picked mon.noname-a con 0x7fe84806ca30 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8480a3370 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe84813c8c0 0x7fe84813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): picked mon.noname-c con 0x7fe84813c8c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8480a2f20 con 0x7fe8480a3370 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fe8480a1b50 con 0x7fe84806ca30 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe84806ce10 con 0x7fe84813c8c0 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84e977640 10 monclient(hunting): authenticate will time out at 938.403225s 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe83f7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fe8480a3370 0x7fe8480a3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:51206/0 (socket says 172.21.15.79:51206) 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe83f7fe640 1 -- 172.21.15.79:0/4072273914 learned_addr learned my addr 172.21.15.79:0/4072273914 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:44.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe847fff640 1 --2- 172.21.15.79:0/4072273914 >> v2:172.21.15.79:3300/0 conn(0x7fe84806ca30 0x7fe8480a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe847fff640 10 monclient(hunting): get_auth_request con 0x7fe84806ca30 auth_method 0 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe847fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe847fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 1 --2- 172.21.15.79:0/4072273914 >> v2:172.21.15.204:3300/0 conn(0x7fe84813c8c0 0x7fe84813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): get_auth_request con 0x7fe84813c8c0 auth_method 0 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.209+0000 7fe84cf14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 10 monclient(hunting): handle_auth_done global_id 4216 payload 274 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 1 -- 172.21.15.79:0/4072273914 >> v2:172.21.15.79:3300/0 conn(0x7fe84806ca30 msgr2=0x7fe8480a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 1 --2- 172.21.15.79:0/4072273914 >> v2:172.21.15.79:3300/0 conn(0x7fe84806ca30 0x7fe8480a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 1 -- 172.21.15.79:0/4072273914 >> v1:172.21.15.79:6789/0 conn(0x7fe8480a3370 legacy=0x7fe8480a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.213+0000 7fe84cf14640 1 -- 172.21.15.79:0/4072273914 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8480a5860 con 0x7fe84813c8c0 2024-04-23T16:39:44.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 1 -- 172.21.15.79:0/4072273914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe84813c8c0 msgr2=0x7fe84813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 1 --2- 172.21.15.79:0/4072273914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe84813c8c0 0x7fe84813ecb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8480a7230 tx=0x7fe84001c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51f226e699e88cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 1 -- 172.21.15.79:0/4072273914 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8400a6710 con 0x7fe84813c8c0 2024-04-23T16:39:44.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: dump: 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.226 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 5 monclient: authenticate success, global_id 4216 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 1 -- 172.21.15.79:0/4072273914 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8400a68b0 con 0x7fe84813c8c0 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 4 set_mon_vals no callback set 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 1 -- 172.21.15.79:0/4072273914 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8400a6c80 con 0x7fe84813c8c0 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe8477fe640 10 monclient: dump: 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: shutdown 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- 172.21.15.79:0/4072273914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe84813c8c0 msgr2=0x7fe84813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 --2- 172.21.15.79:0/4072273914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe84813c8c0 0x7fe84813ecb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8480a7230 tx=0x7fe84001c840 comp rx=0 tx=0).stop 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- 172.21.15.79:0/4072273914 shutdown_connections 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 --2- 172.21.15.79:0/4072273914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe84813c8c0 0x7fe84813ecb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 --2- 172.21.15.79:0/4072273914 >> v2:172.21.15.79:3300/0 conn(0x7fe84806ca30 0x7fe8480a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- 172.21.15.79:0/4072273914 >> 172.21.15.79:0/4072273914 conn(0x7fe8480aa3e0 msgr2=0x7fe8480a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- 172.21.15.79:0/4072273914 shutdown_connections 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- 172.21.15.79:0/4072273914 wait complete. 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: monmap: 2024-04-23T16:39:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 Processor -- start 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- start start 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: init 2024-04-23T16:39:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 0x7fe8481cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): picked mon.noname-c con 0x7fe84806ca30 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8480a3370 0x7fe84813b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): picked mon.noname-b con 0x7fe8480a3370 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 0x7fe848122110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): picked mon.noname-a con 0x7fe84813c8c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe848141da0 con 0x7fe84813c8c0 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe848141c20 con 0x7fe8480a3370 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fe848141f20 con 0x7fe84806ca30 2024-04-23T16:39:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84e977640 10 monclient(hunting): authenticate will time out at 938.411685s 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 0x7fe848122110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe847fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 0x7fe8481cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 0x7fe848122110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47722/0 (socket says 172.21.15.79:47722) 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 1 -- 172.21.15.79:0/3885992897 learned_addr learned my addr 172.21.15.79:0/3885992897 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): get_auth_request con 0x7fe84813c8c0 auth_method 0 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe847fff640 10 monclient(hunting): get_auth_request con 0x7fe84806ca30 auth_method 0 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe847fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe847fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.217+0000 7fe84cf14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8480a3370 0x7fe84813b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): get_auth_request con 0x7fe8480a3370 auth_method 0 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe83f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 -- 172.21.15.79:0/3885992897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8480a3370 msgr2=0x7fe84813b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8480a3370 0x7fe84813b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 -- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 msgr2=0x7fe848122110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 0x7fe848122110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 -- 172.21.15.79:0/3885992897 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe848123d60 con 0x7fe84806ca30 2024-04-23T16:39:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84cf14640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 0x7fe848122110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:44.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 -- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 msgr2=0x7fe8481cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe847fff640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 0x7fe8481cf290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe83800d600 tx=0x7fe83800dac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838012800 con 0x7fe84806ca30 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: dump: 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.237 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe838019070 con 0x7fe84806ca30 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838012bc0 con 0x7fe84806ca30 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 5 monclient: authenticate success, global_id 4193 2024-04-23T16:39:44.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 monclient: _renew_subs 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84813e620 con 0x7fe84806ca30 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe846ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 10 monclient: dump: 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.239 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 monclient: _renew_subs 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84813bc00 con 0x7fe84806ca30 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7fe83801e460 con 0x7fe84806ca30 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe848068ae0 con 0x7fe84806ca30 2024-04-23T16:39:44.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.221+0000 7fe8457fa640 1 --2- 172.21.15.79:0/3885992897 >> v2:172.21.15.204:6812/722020384 conn(0x7fe8280363a0 0x7fe828038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.225+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (secure 0 0 0) 0x7fe8380491a0 con 0x7fe84806ca30 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.225+0000 7fe83f7fe640 1 --2- 172.21.15.79:0/3885992897 >> v2:172.21.15.204:6812/722020384 conn(0x7fe8280363a0 0x7fe828038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.225+0000 7fe83f7fe640 10 monclient: get_auth_request con 0x7fe8280363a0 auth_method 0 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.225+0000 7fe83f7fe640 1 --2- 172.21.15.79:0/3885992897 >> v2:172.21.15.204:6812/722020384 conn(0x7fe8280363a0 0x7fe828038860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe830007ca0 tx=0x7fe830007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.229+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe8380480a0 con 0x7fe84806ca30 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.229+0000 7fe8457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:44.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.229+0000 7fe8457fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:44.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.393+0000 7fe84e977640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:44.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.393+0000 7fe84e977640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:44.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.393+0000 7fe84e977640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.393+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe84813bdd0 con 0x7fe84806ca30 2024-04-23T16:39:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe8457fa640 1 -- 172.21.15.79:0/3885992897 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10812 (secure 0 0 0) 0x7fe83801e750 con 0x7fe84806ca30 2024-04-23T16:39:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe8457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:44.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe8457fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:44.401 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:44.401 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":14,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:39:43.697979+0000","last_up_change":"2024-04-23T16:39:41.690496+0000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:39:41.354561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-23T16:39:42.435523+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":987647954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":987647954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":987647954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":987647954}]},"public_addr":"172.21.15.79:6802/987647954","cluster_addr":"172.21.15.79:6805/987647954","heartbeat_back_addr":"172.21.15.79:6810/987647954","heartbeat_front_addr":"172.21.15.79:6808/987647954","state":["exists","up"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3615156451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":3615156451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":3615156451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":3615156451}]},"public_addr":"172.21.15.79:6803/3615156451","cluster_addr":"172.21.15.79:6806/3615156451","heartbeat_back_addr":"172.21.15.79:6811/3615156451","heartbeat_front_addr":"172.21.15.79:6809/3615156451","state":["exists","up"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3678369310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":3678369310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":3678369310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":3678369310}]},"public_addr":"172.21.15.79:6800/3678369310","cluster_addr":"172.21.15.79:6801/3678369310","heartbeat_back_addr":"172.21.15.79:6807/3678369310","heartbeat_front_addr":"172.21.15.79:6804/3678369310","state":["exists","up"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3194378740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":3194378740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":3194378740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3194378740}]},"public_addr":"172.21.15.204:6800/3194378740","cluster_addr":"172.21.15.204:6801/3194378740","heartbeat_back_addr":"172.21.15.204:6803/3194378740","heartbeat_front_addr":"172.21.15.204:6802/3194378740","state":["exists","up"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3419064638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":3419064638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":3419064638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3419064638}]},"public_addr":"172.21.15.204:6804/3419064638","cluster_addr":"172.21.15.204:6805/3419064638","heartbeat_back_addr":"172.21.15.204:6807/3419064638","heartbeat_front_addr":"172.21.15.204:6806/3419064638","state":["exists","up"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":3961528928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":3961528928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":3961528928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3961528928}]},"public_addr":"172.21.15.204:6808/3961528928","cluster_addr":"172.21.15.204:6809/3961528928","heartbeat_back_addr":"172.21.15.204:6811/3961528928","heartbeat_front_addr":"172.21.15.204:6810/3961528928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.690894+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.747794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.664067+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.480934+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.716414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.584524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:39:44.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 >> v2:172.21.15.204:6812/722020384 conn(0x7fe8280363a0 msgr2=0x7fe828038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 --2- 172.21.15.79:0/3885992897 >> v2:172.21.15.204:6812/722020384 conn(0x7fe8280363a0 0x7fe828038860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe830007ca0 tx=0x7fe830007bf0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 10 monclient: shutdown 2024-04-23T16:39:44.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 msgr2=0x7fe8481cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 0x7fe8481cf290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe83800d600 tx=0x7fe83800dac0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 shutdown_connections 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 --2- 172.21.15.79:0/3885992897 >> v2:172.21.15.204:6812/722020384 conn(0x7fe8280363a0 0x7fe828038860 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84813c8c0 0x7fe848122110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8480a3370 0x7fe84813b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 --2- 172.21.15.79:0/3885992897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe84806ca30 0x7fe8481cf290 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 >> 172.21.15.79:0/3885992897 conn(0x7fe8480aa3e0 msgr2=0x7fe84813d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 shutdown_connections 2024-04-23T16:39:44.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.397+0000 7fe84e977640 1 -- 172.21.15.79:0/3885992897 wait complete. 2024-04-23T16:39:44.420 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-23T16:39:44.420 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T16:39:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient: monmap: 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.535273+0000 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.535273+0000 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.534 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 1 Processor -- start 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 1 -- start start 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient: init 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb1c06ca30 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fbb1c0a3a80 0x7fbb1c0a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient(hunting): picked mon.noname-a con 0x7fbb1c0a3a80 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb1c13c8c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1c0a2f20 con 0x7fbb1c06ca30 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1c0a5860 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1c0a1b50 con 0x7fbb1c0a3a80 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.529+0000 7fbb229e2640 10 monclient(hunting): authenticate will time out at 938.723967s 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fbb1c0a3a80 0x7fbb1c0a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fbb1c0a3a80 0x7fbb1c0a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47732/0 (socket says 172.21.15.79:47732) 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 -- 172.21.15.79:0/3487137577 learned_addr learned my addr 172.21.15.79:0/3487137577 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): get_auth_request con 0x7fbb1c0a3a80 auth_method 0 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb1bfff640 1 -- 172.21.15.79:0/3487137577 >> v1:172.21.15.79:6789/0 conn(0x7fbb1c06ca30 legacy=0x7fbb1c0a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-04-23T16:39:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 -- 172.21.15.79:0/3487137577 >> v1:172.21.15.204:6789/0 conn(0x7fbb1c13c8c0 legacy=0x7fbb1c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 -- 172.21.15.79:0/3487137577 >> v1:172.21.15.79:6789/0 conn(0x7fbb1c06ca30 legacy=0x7fbb1c0a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 monclient: found mon.noname-a 2024-04-23T16:39:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.533+0000 7fbb13fff640 1 -- 172.21.15.79:0/3487137577 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c0a1590 con 0x7fbb1c0a3a80 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb13fff640 1 -- 172.21.15.79:0/3487137577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c0a3a80 msgr2=0x7fbb1c0a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb13fff640 1 --2- 172.21.15.79:0/3487137577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c0a3a80 0x7fbb1c0a2830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbb040098f0 tx=0x7fbb0401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d8c1e003e06629 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 1 -- 172.21.15.79:0/3487137577 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040b0070 con 0x7fbb1c0a3a80 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: dump: 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 1 -- 172.21.15.79:0/3487137577 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb040a97f0 con 0x7fbb1c0a3a80 2024-04-23T16:39:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 1 -- 172.21.15.79:0/3487137577 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040a9bc0 con 0x7fbb1c0a3a80 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb1b7fe640 10 monclient: dump: 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.546 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 5 monclient: authenticate success, global_id 4212 2024-04-23T16:39:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 4 set_mon_vals no callback set 2024-04-23T16:39:44.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 10 monclient: shutdown 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 1 -- 172.21.15.79:0/3487137577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c0a3a80 msgr2=0x7fbb1c0a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 1 --2- 172.21.15.79:0/3487137577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c0a3a80 0x7fbb1c0a2830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbb040098f0 tx=0x7fbb0401c4f0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 1 -- 172.21.15.79:0/3487137577 shutdown_connections 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 1 --2- 172.21.15.79:0/3487137577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c0a3a80 0x7fbb1c0a2830 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 1 -- 172.21.15.79:0/3487137577 >> 172.21.15.79:0/3487137577 conn(0x7fbb1c0aa3e0 msgr2=0x7fbb1c0a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.537+0000 7fbb229e2640 1 -- 172.21.15.79:0/3487137577 shutdown_connections 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 -- 172.21.15.79:0/3487137577 wait complete. 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient: monmap: 2024-04-23T16:39:44.551 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 Processor -- start 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 -- start start 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient: init 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c06ca30 0x7fbb1c1cf320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): picked mon.noname-b con 0x7fbb1c06ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbb1c0a3a80 0x7fbb1c137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): picked mon.noname-c con 0x7fbb1c0a3a80 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 0x7fbb1c09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): picked mon.noname-a con 0x7fbb1c13c8c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c142e00 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c0a18b0 con 0x7fbb1c06ca30 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c142f80 con 0x7fbb1c0a3a80 2024-04-23T16:39:44.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb229e2640 10 monclient(hunting): authenticate will time out at 938.733631s 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 0x7fbb1c09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbb1c0a3a80 0x7fbb1c137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 0x7fbb1c09d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47746/0 (socket says 172.21.15.79:47746) 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 -- 172.21.15.79:0/730635411 learned_addr learned my addr 172.21.15.79:0/730635411 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): get_auth_request con 0x7fbb1c13c8c0 auth_method 0 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): get_auth_request con 0x7fbb1c0a3a80 auth_method 0 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 -- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbb1c0a3a80 msgr2=0x7fbb1c137760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbb1c0a3a80 0x7fbb1c137760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 -- 172.21.15.79:0/730635411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c06ca30 msgr2=0x7fbb1c1cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c06ca30 0x7fbb1c1cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 monclient: found mon.noname-a 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.541+0000 7fbb20f7f640 1 -- 172.21.15.79:0/730635411 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c13f7e0 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb20f7f640 1 -- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 msgr2=0x7fbb1c09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb20f7f640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 0x7fbb1c09d9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c780 tx=0x7fbb1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1400ac00 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: dump: 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 5 monclient: authenticate success, global_id 4215 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb1400ada0 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 monclient: _renew_subs 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c120840 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14004bb0 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 10 monclient: dump: 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.558 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb1affd640 4 set_mon_vals no callback set 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 monclient: _renew_subs 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c09f500 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1c068ae0 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7fbb14005030 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 1 --2- 172.21.15.79:0/730635411 >> v2:172.21.15.204:6812/722020384 conn(0x7fbafc0363d0 0x7fbafc038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (secure 0 0 0) 0x7fbb140490f0 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb1bfff640 1 --2- 172.21.15.79:0/730635411 >> v2:172.21.15.204:6812/722020384 conn(0x7fbafc0363d0 0x7fbafc038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.545+0000 7fbb1bfff640 10 monclient: get_auth_request con 0x7fbafc0363d0 auth_method 0 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.549+0000 7fbb1bfff640 1 --2- 172.21.15.79:0/730635411 >> v2:172.21.15.204:6812/722020384 conn(0x7fbafc0363d0 0x7fbafc038890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c007fd0 tx=0x7fbb0c0079c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.549+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbb14020070 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.549+0000 7fbb197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.549+0000 7fbb197fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:44.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb229e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:44.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb229e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:44.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb229e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb1c072040 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb197fa640 1 -- 172.21.15.79:0/730635411 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10799 (secure 0 0 0) 0x7fbb1c13f7e0 con 0x7fbb1c13c8c0 2024-04-23T16:39:44.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:44.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.717+0000 7fbb197fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:44.724 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:44.724 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":15,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:39:44.700457+0000","last_up_change":"2024-04-23T16:39:41.690496+0000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:39:41.354561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-23T16:39:42.435523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":987647954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":987647954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":987647954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":987647954}]},"public_addr":"172.21.15.79:6802/987647954","cluster_addr":"172.21.15.79:6805/987647954","heartbeat_back_addr":"172.21.15.79:6810/987647954","heartbeat_front_addr":"172.21.15.79:6808/987647954","state":["exists","up"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3615156451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":3615156451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":3615156451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":3615156451}]},"public_addr":"172.21.15.79:6803/3615156451","cluster_addr":"172.21.15.79:6806/3615156451","heartbeat_back_addr":"172.21.15.79:6811/3615156451","heartbeat_front_addr":"172.21.15.79:6809/3615156451","state":["exists","up"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3678369310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":3678369310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":3678369310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":3678369310}]},"public_addr":"172.21.15.79:6800/3678369310","cluster_addr":"172.21.15.79:6801/3678369310","heartbeat_back_addr":"172.21.15.79:6807/3678369310","heartbeat_front_addr":"172.21.15.79:6804/3678369310","state":["exists","up"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3194378740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":3194378740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":3194378740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3194378740}]},"public_addr":"172.21.15.204:6800/3194378740","cluster_addr":"172.21.15.204:6801/3194378740","heartbeat_back_addr":"172.21.15.204:6803/3194378740","heartbeat_front_addr":"172.21.15.204:6802/3194378740","state":["exists","up"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3419064638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":3419064638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":3419064638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3419064638}]},"public_addr":"172.21.15.204:6804/3419064638","cluster_addr":"172.21.15.204:6805/3419064638","heartbeat_back_addr":"172.21.15.204:6807/3419064638","heartbeat_front_addr":"172.21.15.204:6806/3419064638","state":["exists","up"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":3961528928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":3961528928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":3961528928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3961528928}]},"public_addr":"172.21.15.204:6808/3961528928","cluster_addr":"172.21.15.204:6809/3961528928","heartbeat_back_addr":"172.21.15.204:6811/3961528928","heartbeat_front_addr":"172.21.15.204:6810/3961528928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.690894+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.747794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.664067+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.480934+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.716414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.584524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 >> v2:172.21.15.204:6812/722020384 conn(0x7fbafc0363d0 msgr2=0x7fbafc038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 --2- 172.21.15.79:0/730635411 >> v2:172.21.15.204:6812/722020384 conn(0x7fbafc0363d0 0x7fbafc038890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c007fd0 tx=0x7fbb0c0079c0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 10 monclient: shutdown 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 msgr2=0x7fbb1c09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 0x7fbb1c09d9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c780 tx=0x7fbb1400cc40 comp rx=0 tx=0).stop 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 shutdown_connections 2024-04-23T16:39:44.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 --2- 172.21.15.79:0/730635411 >> v2:172.21.15.204:6812/722020384 conn(0x7fbafc0363d0 0x7fbafc038890 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb1c13c8c0 0x7fbb1c09d9b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbb1c0a3a80 0x7fbb1c137760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 --2- 172.21.15.79:0/730635411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c06ca30 0x7fbb1c1cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 >> 172.21.15.79:0/730635411 conn(0x7fbb1c0aa3e0 msgr2=0x7fbb1c0a9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 shutdown_connections 2024-04-23T16:39:44.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.721+0000 7fbb229e2640 1 -- 172.21.15.79:0/730635411 wait complete. 2024-04-23T16:39:44.743 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-23T16:39:44.743 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-23T16:39:44.744 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-23T16:39:44.744 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-23T16:39:44.744 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-23T16:39:44.745 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-23T16:39:44.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient: monmap: 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.863624+0000 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.863624+0000 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.863 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.864 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.864 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.864 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 Processor -- start 2024-04-23T16:39:44.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 -- start start 2024-04-23T16:39:44.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient: init 2024-04-23T16:39:44.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b880a3600 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3b880a39e0 0x7f3b8813c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): picked mon.noname-c con 0x7f3b880a39e0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3b8813ce70 0x7f3b8813f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): picked mon.noname-a con 0x7f3b8813ce70 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b881383a0 con 0x7f3b880a3600 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f3b880a0f10 con 0x7f3b8813ce70 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3b880a5d40 con 0x7f3b880a39e0 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3b8813ce70 0x7f3b8813f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3b8813ce70 0x7f3b8813f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47750/0 (socket says 172.21.15.79:47750) 2024-04-23T16:39:44.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 1 -- 172.21.15.79:0/3845022625 learned_addr learned my addr 172.21.15.79:0/3845022625 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8efbf640 10 monclient(hunting): authenticate will time out at 939.051686s 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): get_auth_request con 0x7f3b8813ce70 auth_method 0 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b87fff640 1 --2- 172.21.15.79:0/3845022625 >> v2:172.21.15.204:3300/0 conn(0x7f3b880a39e0 0x7f3b8813c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b87fff640 10 monclient(hunting): get_auth_request con 0x7f3b880a39e0 auth_method 0 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b87fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.857+0000 7f3b8d55c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 1 -- 172.21.15.79:0/3845022625 >> v2:172.21.15.204:3300/0 conn(0x7f3b880a39e0 msgr2=0x7f3b8813c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 1 --2- 172.21.15.79:0/3845022625 >> v2:172.21.15.204:3300/0 conn(0x7f3b880a39e0 0x7f3b8813c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 1 -- 172.21.15.79:0/3845022625 >> v1:172.21.15.204:6789/0 conn(0x7f3b880a3600 legacy=0x7f3b88137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 1 monclient: found mon.noname-a 2024-04-23T16:39:44.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f3b8d55c640 1 -- 172.21.15.79:0/3845022625 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b880a1600 con 0x7f3b8813ce70 2024-04-23T16:39:44.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient: monmap: 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.866830+0000 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.866830+0000 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.873 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 Processor -- start 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 -- start start 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient: init 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-23T16:39:44.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff40a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff40a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0ff4137c90 0x7f0ff40a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff4137c90 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ff4138090 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff40a5d40 con 0x7f0ff4138090 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f0ff40a0f10 con 0x7f0ff4137c90 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff40a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff40a3f20 con 0x7f0ff40a3600 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff40a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47126/0 (socket says 172.21.15.79:47126) 2024-04-23T16:39:44.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ffa6f9640 10 monclient(hunting): authenticate will time out at 939.054860s 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 -- 172.21.15.79:0/2991764358 learned_addr learned my addr 172.21.15.79:0/2991764358 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): get_auth_request con 0x7f0ff40a3600 auth_method 0 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 -- 172.21.15.79:0/2991764358 >> v2:172.21.15.79:3300/0 conn(0x7f0ff4137c90 msgr2=0x7f0ff40a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 --2- 172.21.15.79:0/2991764358 >> v2:172.21.15.79:3300/0 conn(0x7f0ff4137c90 0x7f0ff40a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 -- 172.21.15.79:0/2991764358 >> v1:172.21.15.204:6789/0 conn(0x7f0ff4138090 legacy=0x7f0ff40a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 monclient: found mon.noname-b 2024-04-23T16:39:44.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.861+0000 7f0ff3fff640 1 -- 172.21.15.79:0/2991764358 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff40a1600 con 0x7f0ff40a3600 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8d55c640 1 -- 172.21.15.79:0/3845022625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 msgr2=0x7f3b8813f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8d55c640 1 --2- 172.21.15.79:0/3845022625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 0x7f3b8813f260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b88137af0 tx=0x7f3b8001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e813a48ca2e5364d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 1 -- 172.21.15.79:0/3845022625 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b800a5570 con 0x7f3b8813ce70 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: dump: 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.879 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 1 -- 172.21.15.79:0/3845022625 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b800a5710 con 0x7f3b8813ce70 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 1 -- 172.21.15.79:0/3845022625 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b800a5ae0 con 0x7f3b8813ce70 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b877fe640 10 monclient: dump: 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 5 monclient: authenticate success, global_id 4218 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 4 set_mon_vals no callback set 2024-04-23T16:39:44.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 10 monclient: shutdown 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 -- 172.21.15.79:0/3845022625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 msgr2=0x7f3b8813f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/3845022625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 0x7f3b8813f260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b88137af0 tx=0x7f3b8001c890 comp rx=0 tx=0).stop 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 -- 172.21.15.79:0/3845022625 shutdown_connections 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/3845022625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 0x7f3b8813f260 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/3845022625 >> v2:172.21.15.204:3300/0 conn(0x7f3b880a39e0 0x7f3b8813c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 -- 172.21.15.79:0/3845022625 >> 172.21.15.79:0/3845022625 conn(0x7f3b880aa3e0 msgr2=0x7f3b880a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 -- 172.21.15.79:0/3845022625 shutdown_connections 2024-04-23T16:39:44.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 1 -- 172.21.15.79:0/3845022625 wait complete. 2024-04-23T16:39:44.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff3fff640 1 -- 172.21.15.79:0/2991764358 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 msgr2=0x7f0ff40a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff3fff640 1 --2- 172.21.15.79:0/2991764358 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff40a39e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ff406d050 tx=0x7f0fe401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355533bd841ba748 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 1 -- 172.21.15.79:0/2991764358 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40b0070 con 0x7f0ff40a3600 2024-04-23T16:39:44.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:44.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: dump: 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 1 -- 172.21.15.79:0/2991764358 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe40ab600 con 0x7f0ff40a3600 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 1 -- 172.21.15.79:0/2991764358 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40ab9d0 con 0x7f0ff40a3600 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ff2ffd640 10 monclient: dump: 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.886 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ffa6f9640 5 monclient: authenticate success, global_id 4199 2024-04-23T16:39:44.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ffa6f9640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f0ffa6f9640 4 set_mon_vals no callback set 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.865+0000 7f3b8efbf640 10 monclient: monmap: 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 Processor -- start 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 -- start start 2024-04-23T16:39:44.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient: init 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient: shutdown 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/2991764358 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 msgr2=0x7f0ff40a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 --2- 172.21.15.79:0/2991764358 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff40a39e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ff406d050 tx=0x7f0fe401c3e0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/2991764358 shutdown_connections 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 --2- 172.21.15.79:0/2991764358 >> v2:172.21.15.79:3300/0 conn(0x7f0ff4137c90 0x7f0ff40a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 --2- 172.21.15.79:0/2991764358 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff40a39e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/2991764358 >> 172.21.15.79:0/2991764358 conn(0x7f0ff40aa3e0 msgr2=0x7f0ff40a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/2991764358 shutdown_connections 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 0x7f3b8809fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): picked mon.noname-c con 0x7f3b880a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 0x7f3b8809fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 0x7f3b8809fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47134/0 (socket says 172.21.15.79:47134) 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 -- 172.21.15.79:0/2980484250 learned_addr learned my addr 172.21.15.79:0/2980484250 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b880a39e0 0x7f3b880a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): picked mon.noname-b con 0x7f3b880a39e0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 0x7f3b8809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): picked mon.noname-a con 0x7f3b8813ce70 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8d55c640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 0x7f3b8809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3b88142140 con 0x7f3b8813ce70 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3b88141fc0 con 0x7f3b880a39e0 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f3b881422c0 con 0x7f3b880a3600 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b880a39e0 0x7f3b880a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8efbf640 10 monclient(hunting): authenticate will time out at 939.062003s 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): get_auth_request con 0x7f3b880a3600 auth_method 0 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): get_auth_request con 0x7f3b880a39e0 auth_method 0 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/2991764358 wait complete. 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient: monmap: 2024-04-23T16:39:44.893 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 Processor -- start 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8d55c640 10 monclient(hunting): get_auth_request con 0x7f3b8813ce70 auth_method 0 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8d55c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8d55c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8d55c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 -- 172.21.15.79:0/2980484250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b880a39e0 msgr2=0x7f3b880a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b880a39e0 0x7f3b880a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 -- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 msgr2=0x7f3b8809ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b8813ce70 0x7f3b8809ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f3b8cd5b640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b880a80d0 con 0x7f3b880a3600 2024-04-23T16:39:44.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- start start 2024-04-23T16:39:44.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient: init 2024-04-23T16:39:44.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff41dc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff40a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ff4137c90 0x7f0ff41dcab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff4137c90 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ff4138090 0x7f0ff41dcff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff4138090 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff41422f0 con 0x7f0ff4137c90 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff4142170 con 0x7f0ff4138090 2024-04-23T16:39:44.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff4142470 con 0x7f0ff40a3600 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ffa6f9640 10 monclient(hunting): authenticate will time out at 939.062948s 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff41dc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff41dc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47138/0 (socket says 172.21.15.79:47138) 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ff4137c90 0x7f0ff41dcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 -- 172.21.15.79:0/17917467 learned_addr learned my addr 172.21.15.79:0/17917467 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): get_auth_request con 0x7f0ff40a3600 auth_method 0 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): get_auth_request con 0x7f0ff4137c90 auth_method 0 2024-04-23T16:39:44.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 1 --2- 172.21.15.79:0/17917467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ff4138090 0x7f0ff41dcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): get_auth_request con 0x7f0ff4138090 auth_method 0 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff8c96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 -- 172.21.15.79:0/17917467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ff4138090 msgr2=0x7f0ff41dcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 --2- 172.21.15.79:0/17917467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ff4138090 0x7f0ff41dcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 -- 172.21.15.79:0/17917467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ff4137c90 msgr2=0x7f0ff41dcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 --2- 172.21.15.79:0/17917467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ff4137c90 0x7f0ff41dcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.869+0000 7f0ff3fff640 1 -- 172.21.15.79:0/17917467 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff40a40e0 con 0x7f0ff40a3600 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8cd5b640 1 -- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 msgr2=0x7f3b8809fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8cd5b640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 0x7f3b8809fc70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00c450 tx=0x7f3b7c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c012a40 con 0x7f3b880a3600 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: dump: 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.904 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 5 monclient: authenticate success, global_id 4202 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 10 monclient: _renew_subs 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8809dea0 con 0x7f3b880a3600 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 10 monclient: _renew_subs 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8809cfb0 con 0x7f3b880a3600 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7c017070 con 0x7f3b880a3600 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c012e40 con 0x7f3b880a3600 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b86ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: dump: 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.906 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff3fff640 1 -- 172.21.15.79:0/17917467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 msgr2=0x7f0ff41dc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff3fff640 1 --2- 172.21.15.79:0/17917467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff41dc570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40ae040 tx=0x7f0fe401ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b68000f90 con 0x7f3b880a3600 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40ba020 con 0x7f0ff40a3600 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: dump: 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.908 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe40b6630 con 0x7f0ff40a3600 2024-04-23T16:39:44.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40b0040 con 0x7f0ff40a3600 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff0ff9640 10 monclient: dump: 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.910 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 5 monclient: authenticate success, global_id 4205 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 10 monclient: _renew_subs 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff4134a00 con 0x7f0ff40a3600 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ff27fc640 4 set_mon_vals no callback set 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 10 monclient: _renew_subs 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff4134720 con 0x7f0ff40a3600 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f3b7c01c460 con 0x7f3b880a3600 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.204:6812/722020384 conn(0x7f3b6c036350 0x7f3b6c038810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f3b7c048bb0 con 0x7f3b880a3600 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.79:6802/987647954 conn(0x7f3b6c03be60 0x7f3b6c03e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 --> v2:172.21.15.79:6802/987647954 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b7c00ae50 con 0x7f3b6c03be60 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f3b7c0194e0 con 0x7f3b880a3600 2024-04-23T16:39:44.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b857fa640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-23T16:39:44.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8d55c640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.79:6802/987647954 conn(0x7f3b6c03be60 0x7f3b6c03e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.873+0000 7f3b8d55c640 10 monclient: get_auth_request con 0x7f3b6c03be60 auth_method 0 2024-04-23T16:39:44.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f3b87fff640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.204:6812/722020384 conn(0x7f3b6c036350 0x7f3b6c038810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f3b87fff640 10 monclient: get_auth_request con 0x7f3b6c036350 auth_method 0 2024-04-23T16:39:44.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ffa6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ff4068ae0 con 0x7f0ff40a3600 2024-04-23T16:39:44.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f0fe40bd020 con 0x7f0ff40a3600 2024-04-23T16:39:44.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6812/722020384 conn(0x7f0fe00363a0 0x7f0fe0038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f0fe40e2ca0 con 0x7f0ff40a3600 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6804/3419064638 conn(0x7f0fe003bec0 0x7f0fe003e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 --> v2:172.21.15.204:6804/3419064638 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fe40044a0 con 0x7f0fe003bec0 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f0fe40e9050 con 0x7f0ff40a3600 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff0ff9640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff8c96640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6804/3419064638 conn(0x7f0fe003bec0 0x7f0fe003e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff8c96640 10 monclient: get_auth_request con 0x7f0fe003bec0 auth_method 0 2024-04-23T16:39:44.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f3b87fff640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.204:6812/722020384 conn(0x7f3b6c036350 0x7f3b6c038810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3b8813bba0 tx=0x7f3b74009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff37fe640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6812/722020384 conn(0x7f0fe00363a0 0x7f0fe0038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.877+0000 7f0ff37fe640 10 monclient: get_auth_request con 0x7f0fe00363a0 auth_method 0 2024-04-23T16:39:44.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.881+0000 7f3b8d55c640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.79:6802/987647954 conn(0x7f3b6c03be60 0x7f3b6c03e280 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.881+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== osd.0 v2:172.21.15.79:6802/987647954 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b7c00ae50 con 0x7f3b6c03be60 2024-04-23T16:39:44.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.881+0000 7f0ff37fe640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6812/722020384 conn(0x7f0fe00363a0 0x7f0fe0038860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ff41374e0 tx=0x7f0fe8009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.881+0000 7f0ff8c96640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6804/3419064638 conn(0x7f0fe003bec0 0x7f0fe003e2e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.885+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== osd.4 v2:172.21.15.204:6804/3419064638 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fe40044a0 con 0x7f0fe003bec0 2024-04-23T16:39:44.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 --> v2:172.21.15.79:6802/987647954 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b68002c80 con 0x7f3b6c03be60 2024-04-23T16:39:44.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b857fa640 1 -- 172.21.15.79:0/2980484250 <== osd.0 v2:172.21.15.79:6802/987647954 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3b68002c80 con 0x7f3b6c03be60 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 >> v2:172.21.15.79:6802/987647954 conn(0x7f3b6c03be60 msgr2=0x7f3b6c03e280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.79:6802/987647954 conn(0x7f3b6c03be60 0x7f3b6c03e280 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 >> v2:172.21.15.204:6812/722020384 conn(0x7f3b6c036350 msgr2=0x7f3b6c038810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/2980484250 >> v2:172.21.15.204:6812/722020384 conn(0x7f3b6c036350 0x7f3b6c038810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3b8813bba0 tx=0x7f3b74009190 comp rx=0 tx=0).stop 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 10 monclient: shutdown 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 msgr2=0x7f3b8809fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 --2- 172.21.15.79:0/2980484250 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3b880a3600 0x7f3b8809fc70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00c450 tx=0x7f3b7c00c910 comp rx=0 tx=0).stop 2024-04-23T16:39:44.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8d55c640 1 -- 172.21.15.79:0/2980484250 reap_dead start 2024-04-23T16:39:44.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 shutdown_connections 2024-04-23T16:39:44.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 >> 172.21.15.79:0/2980484250 conn(0x7f3b880aa3e0 msgr2=0x7f3b880a95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 shutdown_connections 2024-04-23T16:39:44.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.897+0000 7f3b8efbf640 1 -- 172.21.15.79:0/2980484250 wait complete. 2024-04-23T16:39:44.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient: monmap: 2024-04-23T16:39:44.921 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.913883+0000 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.913883+0000 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 Processor -- start 2024-04-23T16:39:44.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 -- start start 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient: init 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7ab40a3600 0x7f7ab40a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): picked mon.noname-c con 0x7f7ab40a3600 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab4137c90 0x7f7ab40a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): picked mon.noname-b con 0x7f7ab4137c90 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7ab4138090 0x7f7ab40a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7ab4138090 0x7f7ab40a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7ab4138090 0x7f7ab40a4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47782/0 (socket says 172.21.15.79:47782) 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 1 -- 172.21.15.79:0/685204222 learned_addr learned my addr 172.21.15.79:0/685204222 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): picked mon.noname-a con 0x7f7ab4138090 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/685204222 >> v2:172.21.15.204:3300/0 conn(0x7f7ab40a3600 0x7f7ab40a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f7ab40a0f10 con 0x7f7ab4138090 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7ab40a3f20 con 0x7f7ab40a3600 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab40a5d40 con 0x7f7ab4137c90 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab9844640 10 monclient(hunting): authenticate will time out at 939.102122s 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): get_auth_request con 0x7f7ab40a3600 auth_method 0 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 1 --2- 172.21.15.79:0/685204222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab4137c90 0x7f7ab40a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): get_auth_request con 0x7f7ab4137c90 auth_method 0 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): get_auth_request con 0x7f7ab4138090 auth_method 0 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_done global_id 4231 payload 274 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/685204222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab4137c90 msgr2=0x7f7ab40a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/685204222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab4137c90 0x7f7ab40a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/685204222 >> v2:172.21.15.79:3300/0 conn(0x7f7ab4138090 msgr2=0x7f7ab40a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/685204222 >> v2:172.21.15.79:3300/0 conn(0x7f7ab4138090 0x7f7ab40a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:44.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.909+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/685204222 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab40a1600 con 0x7f7ab40a3600 2024-04-23T16:39:44.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 --> v2:172.21.15.204:6804/3419064638 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ff4132ff0 con 0x7f0fe003bec0 2024-04-23T16:39:44.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ff0ff9640 1 -- 172.21.15.79:0/17917467 <== osd.4 v2:172.21.15.204:6804/3419064638 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ff4132ff0 con 0x7f0fe003bec0 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6804/3419064638 conn(0x7f0fe003bec0 msgr2=0x7f0fe003e2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6804/3419064638 conn(0x7f0fe003bec0 0x7f0fe003e2e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6812/722020384 conn(0x7f0fe00363a0 msgr2=0x7f0fe0038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 --2- 172.21.15.79:0/17917467 >> v2:172.21.15.204:6812/722020384 conn(0x7f0fe00363a0 0x7f0fe0038860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ff41374e0 tx=0x7f0fe8009110 comp rx=0 tx=0).stop 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 10 monclient: shutdown 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 msgr2=0x7f0ff41dc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 --2- 172.21.15.79:0/17917467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ff40a3600 0x7f0ff41dc570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40ae040 tx=0x7f0fe401ac20 comp rx=0 tx=0).stop 2024-04-23T16:39:44.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ff8c96640 1 -- 172.21.15.79:0/17917467 reap_dead start 2024-04-23T16:39:44.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 shutdown_connections 2024-04-23T16:39:44.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 >> 172.21.15.79:0/17917467 conn(0x7f0ff40aa3e0 msgr2=0x7f0ff40a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 shutdown_connections 2024-04-23T16:39:44.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.913+0000 7f0ffa6f9640 1 -- 172.21.15.79:0/17917467 wait complete. 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/685204222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab40a3600 msgr2=0x7f7ab40a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/685204222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab40a3600 0x7f7ab40a39e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40a7100 tx=0x7f7aa401c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=451fc35ebc292251 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 1 -- 172.21.15.79:0/685204222 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40b0070 con 0x7f7ab40a3600 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: dump: 2024-04-23T16:39:44.933 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 1 -- 172.21.15.79:0/685204222 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa40ab600 con 0x7f7ab40a3600 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 1 -- 172.21.15.79:0/685204222 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40ab9d0 con 0x7f7ab40a3600 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab1ffb640 10 monclient: dump: 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.935 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 5 monclient: authenticate success, global_id 4231 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 4 set_mon_vals no callback set 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: shutdown 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab40a3600 msgr2=0x7f7ab40a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- 172.21.15.79:0/685204222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab40a3600 0x7f7ab40a39e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40a7100 tx=0x7f7aa401c340 comp rx=0 tx=0).stop 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 shutdown_connections 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- 172.21.15.79:0/685204222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab4137c90 0x7f7ab40a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- 172.21.15.79:0/685204222 >> v2:172.21.15.79:3300/0 conn(0x7f7ab4138090 0x7f7ab40a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- 172.21.15.79:0/685204222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab40a3600 0x7f7ab40a39e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 >> 172.21.15.79:0/685204222 conn(0x7f7ab40aa3e0 msgr2=0x7f7ab40a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 shutdown_connections 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- 172.21.15.79:0/685204222 wait complete. 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: monmap: 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:44.937 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 Processor -- start 2024-04-23T16:39:44.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- start start 2024-04-23T16:39:44.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: init 2024-04-23T16:39:44.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f4da475f640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f4da475f640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f4da475f640 10 monclient: monmap: 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.925415+0000 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.925415+0000 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.939 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:44.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 0x7f7ab41cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): picked mon.noname-c con 0x7f7ab40a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab4137c90 0x7f7ab41346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): picked mon.noname-b con 0x7f7ab4137c90 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ab4138090 0x7f7ab413a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): picked mon.noname-a con 0x7f7ab4138090 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab4141370 con 0x7f7ab4138090 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab40a5670 con 0x7f7ab4137c90 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab41414f0 con 0x7f7ab40a3600 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.917+0000 7f7ab9844640 10 monclient(hunting): authenticate will time out at 939.111968s 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 0x7f7ab41cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ab4138090 0x7f7ab413a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 0x7f7ab41cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47152/0 (socket says 172.21.15.79:47152) 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): get_auth_request con 0x7f7ab4138090 auth_method 0 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/2385506209 learned_addr learned my addr 172.21.15.79:0/2385506209 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): get_auth_request con 0x7f7ab40a3600 auth_method 0 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab27fc640 1 --2- 172.21.15.79:0/2385506209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab4137c90 0x7f7ab41346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab27fc640 10 monclient(hunting): get_auth_request con 0x7f7ab4137c90 auth_method 0 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab27fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-04-23T16:39:44.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/2385506209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab4137c90 msgr2=0x7f7ab41346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/2385506209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ab4137c90 0x7f7ab41346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ab4138090 msgr2=0x7f7ab413a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ab4138090 0x7f7ab413a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/2385506209 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab413df40 con 0x7f7ab40a3600 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab37fe640 1 --2- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ab4138090 0x7f7ab413a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:44.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 Processor -- start 2024-04-23T16:39:44.946 INFO:teuthology.orchestra.run.smithi079.stdout:51539607554 2024-04-23T16:39:44.947 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T16:39:44.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 -- start start 2024-04-23T16:39:44.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient: init 2024-04-23T16:39:44.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f4d9c06ecc0 0x7f4d9c06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): picked mon.noname-a con 0x7f4d9c06ecc0 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4d9c0734a0 0x7f4d9c06f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): picked mon.noname-c con 0x7f4d9c0734a0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c134360 0x7f4d9c1347d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): picked mon.noname-b con 0x7f4d9c134360 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f4d9c0a3450 con 0x7f4d9c06ecc0 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f4d9c0a2d50 con 0x7f4d9c0734a0 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9c070be0 con 0x7f4d9c134360 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da475f640 10 monclient(hunting): authenticate will time out at 939.113688s 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f4d9c06ecc0 0x7f4d9c06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f4d9c06ecc0 0x7f4d9c06f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47804/0 (socket says 172.21.15.79:47804) 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 -- 172.21.15.79:0/2229638823 learned_addr learned my addr 172.21.15.79:0/2229638823 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): get_auth_request con 0x7f4d9c06ecc0 auth_method 0 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da1cfa640 1 --2- 172.21.15.79:0/2229638823 >> v2:172.21.15.204:3300/0 conn(0x7f4d9c0734a0 0x7f4d9c06f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 -- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c134360 msgr2=0x7f4d9c1347d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da2cfc640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c134360 0x7f4d9c1347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c134360 0x7f4d9c1347d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 -- 172.21.15.79:0/2229638823 >> v2:172.21.15.204:3300/0 conn(0x7f4d9c0734a0 msgr2=0x7f4d9c06f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 --2- 172.21.15.79:0/2229638823 >> v2:172.21.15.204:3300/0 conn(0x7f4d9c0734a0 0x7f4d9c06f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 monclient: found mon.noname-a 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da24fb640 1 -- 172.21.15.79:0/2229638823 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9c071e60 con 0x7f4d9c06ecc0 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da1cfa640 10 monclient: get_auth_request con 0x7f4d9c0734a0 auth_method 0 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da1cfa640 1 --2- 172.21.15.79:0/2229638823 >> v2:172.21.15.204:3300/0 conn(0x7f4d9c0734a0 0x7f4d9c06f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da2cfc640 10 monclient: get_auth_request con 0x7f4d9c134360 auth_method 0 2024-04-23T16:39:44.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f4da2cfc640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c134360 0x7f4d9c1347d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fcad9a6c640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fcad9a6c640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fcad9a6c640 10 monclient: monmap: 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.928448+0000 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.928448+0000 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.954 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.955 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.955 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.955 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.955 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 -- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 msgr2=0x7f7ab41cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab2ffd640 1 --2- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 0x7f7ab41cf8f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7aa401cbe0 tx=0x7f7aa40190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fd25adc1640 10 monclient: get_monmap_and_config 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fd25adc1640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fd25adc1640 10 monclient: monmap: 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:44.929049+0000 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:44.929049+0000 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:44.956 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40b0040 con 0x7f7ab40a3600 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: dump: 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.957 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa40b9070 con 0x7f7ab40a3600 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40aaa10 con 0x7f7ab40a3600 2024-04-23T16:39:44.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fcad9a6c640 1 Processor -- start 2024-04-23T16:39:44.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:44.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7a9bfff640 10 monclient: dump: 2024-04-23T16:39:44.960 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.960 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.960 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.961 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.961 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.961 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.961 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.961 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.961 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 5 monclient: authenticate success, global_id 4211 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 10 monclient: _renew_subs 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab413e220 con 0x7f7ab40a3600 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab17fa640 4 set_mon_vals no callback set 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 10 monclient: _renew_subs 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab40a8d10 con 0x7f7ab40a3600 2024-04-23T16:39:44.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fcad9a6c640 1 -- start start 2024-04-23T16:39:44.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.921+0000 7fcad9a6c640 10 monclient: init 2024-04-23T16:39:44.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 Processor -- start 2024-04-23T16:39:44.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 -- start start 2024-04-23T16:39:44.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient: init 2024-04-23T16:39:44.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-23T16:39:44.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcad4137a10 0x7fcad4137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): picked mon.noname-c con 0x7fcad4137a10 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcad409fe10 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcad40a2be0 0x7fcad413f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): picked mon.noname-a con 0x7fcad40a2be0 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad4138330 con 0x7fcad409fe10 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fcad40a09b0 con 0x7fcad40a2be0 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcad40a0c90 con 0x7fcad4137a10 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad9a6c640 10 monclient(hunting): authenticate will time out at 939.116657s 2024-04-23T16:39:44.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-23T16:39:44.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcad4137a10 0x7fcad4137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd2540a3600 0x7fd2540a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7ab9844640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ab413cf00 con 0x7f7ab40a3600 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcad4137a10 0x7fcad4137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.79:46490/0 (socket says 172.21.15.79:46490) 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 -- 172.21.15.79:0/3349348064 learned_addr learned my addr 172.21.15.79:0/3349348064 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): get_auth_request con 0x7fcad4137a10 auth_method 0 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): picked mon.noname-c con 0x7fd2540a3600 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:44.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd254137c90 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fd254138090 0x7fd2540a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): picked mon.noname-a con 0x7fd254138090 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2540a5d40 con 0x7fd254137c90 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fd2540a0f10 con 0x7fd254138090 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd2540a3f20 con 0x7fd2540a3600 2024-04-23T16:39:44.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad3fff640 1 --2- 172.21.15.79:0/3349348064 >> v2:172.21.15.79:3300/0 conn(0x7fcad40a2be0 0x7fcad413f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd2540a3600 0x7fd2540a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd25adc1640 10 monclient(hunting): authenticate will time out at 939.117250s 2024-04-23T16:39:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd2540a3600 0x7fd2540a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.79:46498/0 (socket says 172.21.15.79:46498) 2024-04-23T16:39:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 -- 172.21.15.79:0/3187001137 learned_addr learned my addr 172.21.15.79:0/3187001137 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): get_auth_request con 0x7fd2540a3600 auth_method 0 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad3fff640 10 monclient(hunting): get_auth_request con 0x7fcad40a2be0 auth_method 0 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient(hunting): handle_auth_done global_id 4237 payload 274 2024-04-23T16:39:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 -- 172.21.15.79:0/3349348064 >> v2:172.21.15.79:3300/0 conn(0x7fcad40a2be0 msgr2=0x7fcad413f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 --2- 172.21.15.79:0/3349348064 >> v2:172.21.15.79:3300/0 conn(0x7fcad40a2be0 0x7fcad413f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 -- 172.21.15.79:0/3349348064 >> v1:172.21.15.204:6789/0 conn(0x7fcad409fe10 legacy=0x7fcad40a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fcad37fe640 1 -- 172.21.15.79:0/3349348064 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad40a0f70 con 0x7fcad4137a10 2024-04-23T16:39:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient(hunting): handle_auth_done global_id 4240 payload 274 2024-04-23T16:39:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 -- 172.21.15.79:0/3187001137 >> v2:172.21.15.79:3300/0 conn(0x7fd254138090 msgr2=0x7fd2540a4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 --2- 172.21.15.79:0/3187001137 >> v2:172.21.15.79:3300/0 conn(0x7fd254138090 0x7fd2540a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 -- 172.21.15.79:0/3187001137 >> v1:172.21.15.204:6789/0 conn(0x7fd254137c90 legacy=0x7fd2540a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:44.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:44.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7fd258b5d640 1 -- 172.21.15.79:0/3187001137 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2540a1600 con 0x7fd2540a3600 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f7aa40ab4c0 con 0x7f7ab40a3600 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.204:6812/722020384 conn(0x7f7aa00363a0 0x7f7aa0038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f7aa40e3640 con 0x7f7ab40a3600 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.79:6803/3615156451 conn(0x7f7aa003bf30 0x7f7aa003e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 --> v2:172.21.15.79:6803/3615156451 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7aa40ace50 con 0x7f7aa003bf30 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f7aa40e3920 con 0x7f7ab40a3600 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7a9bfff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7ab27fc640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.204:6812/722020384 conn(0x7f7aa00363a0 0x7f7aa0038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7ab37fe640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.79:6803/3615156451 conn(0x7f7aa003bf30 0x7f7aa003e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7ab27fc640 10 monclient: get_auth_request con 0x7f7aa00363a0 auth_method 0 2024-04-23T16:39:44.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f7ab37fe640 10 monclient: get_auth_request con 0x7f7aa003bf30 auth_method 0 2024-04-23T16:39:44.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da24fb640 1 -- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 msgr2=0x7f4d9c06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da24fb640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c06f0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c070900 tx=0x7f4d9801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b40775c1db79651 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 1 -- 172.21.15.79:0/2229638823 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d980b2070 con 0x7f4d9c06ecc0 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: dump: 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.979 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 1 -- 172.21.15.79:0/2229638823 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d980ad680 con 0x7f4d9c06ecc0 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da475f640 5 monclient: authenticate success, global_id 4233 2024-04-23T16:39:44.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da475f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da475f640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da475f640 4 set_mon_vals no callback set 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 1 -- 172.21.15.79:0/2229638823 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d980ada50 con 0x7f4d9c06ecc0 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.925+0000 7f4da14f9640 10 monclient: dump: 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.981 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient: shutdown 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 msgr2=0x7f4d9c06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c06f0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c070900 tx=0x7f4d9801c3e0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- 172.21.15.79:0/2229638823 shutdown_connections 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c134360 0x7f4d9c1347d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- 172.21.15.79:0/2229638823 >> v2:172.21.15.204:3300/0 conn(0x7f4d9c0734a0 0x7f4d9c06f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- 172.21.15.79:0/2229638823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c06f0a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- 172.21.15.79:0/2229638823 >> 172.21.15.79:0/2229638823 conn(0x7f4d9c0a36a0 msgr2=0x7f4d9c0a3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:44.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- 172.21.15.79:0/2229638823 shutdown_connections 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- 172.21.15.79:0/2229638823 wait complete. 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient: build_initial_monmap 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient: monmap: 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:44.983 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f7ab27fc640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.204:6812/722020384 conn(0x7f7aa00363a0 0x7f7aa0038860 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7ab413bb30 tx=0x7f7aa8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 Processor -- start 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- start start 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient: init 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c133790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): picked mon.noname-a con 0x7f4d9c06ecc0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c0734a0 0x7f4d9c133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): picked mon.noname-c con 0x7f4d9c0734a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:44.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d9c134360 0x7f4d9c11d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): picked mon.noname-b con 0x7f4d9c134360 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9c13e050 con 0x7f4d9c06ecc0 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9c06fd70 con 0x7f4d9c134360 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9c13e1d0 con 0x7f4d9c0734a0 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da475f640 10 monclient(hunting): authenticate will time out at 939.122148s 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c133790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c133790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47826/0 (socket says 172.21.15.79:47826) 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c0734a0 0x7f4d9c133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 1 -- 172.21.15.79:0/1498705707 learned_addr learned my addr 172.21.15.79:0/1498705707 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:44.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): get_auth_request con 0x7f4d9c06ecc0 auth_method 0 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): get_auth_request con 0x7f4d9c0734a0 auth_method 0 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da24fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:44.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da2cfc640 1 --2- 172.21.15.79:0/1498705707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d9c134360 0x7f4d9c11d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da2cfc640 10 monclient(hunting): get_auth_request con 0x7f4d9c134360 auth_method 0 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da2cfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da2cfc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da2cfc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd258b5d640 1 -- 172.21.15.79:0/3187001137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2540a3600 msgr2=0x7fd2540a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd258b5d640 1 --2- 172.21.15.79:0/3187001137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2540a3600 0x7fd2540a39e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2540a7100 tx=0x7fd24801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d18080b05102ebe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 -- 172.21.15.79:0/1498705707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d9c134360 msgr2=0x7f4d9c11d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 --2- 172.21.15.79:0/1498705707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d9c134360 0x7f4d9c11d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 -- 172.21.15.79:0/1498705707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 msgr2=0x7f4d9c133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 --2- 172.21.15.79:0/1498705707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d9c06ecc0 0x7f4d9c133790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient: _finish_hunting 0 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 monclient: found mon.noname-c 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:44.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f4da1cfa640 1 -- 172.21.15.79:0/1498705707 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9c0a0e30 con 0x7f4d9c0734a0 2024-04-23T16:39:44.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad37fe640 1 -- 172.21.15.79:0/3349348064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad4137a10 msgr2=0x7fcad4137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:44.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad37fe640 1 --2- 172.21.15.79:0/3349348064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad4137a10 0x7fcad4137df0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcad40753a0 tx=0x7fcac401c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d63d77656234bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 1 -- 172.21.15.79:0/3187001137 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2480b0070 con 0x7fd2540a3600 2024-04-23T16:39:44.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: dump: 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 5 monclient: authenticate success, global_id 4240 2024-04-23T16:39:44.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:44.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f7ab37fe640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.79:6803/3615156451 conn(0x7f7aa003bf30 0x7f7aa003e350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:44.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 1 -- 172.21.15.79:0/3187001137 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2480ab600 con 0x7fd2540a3600 2024-04-23T16:39:44.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 1 -- 172.21.15.79:0/3187001137 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2480ab9d0 con 0x7fd2540a3600 2024-04-23T16:39:44.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 4 set_mon_vals no callback set 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd2537fe640 10 monclient: dump: 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.994 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 1 -- 172.21.15.79:0/3349348064 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac40b0070 con 0x7fcad4137a10 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: dump: 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.995 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: _finish_auth 0 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: _check_auth_tickets 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 1 -- 172.21.15.79:0/3349348064 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac40ab600 con 0x7fcad4137a10 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:44.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 1 -- 172.21.15.79:0/3349348064 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac40ab9d0 con 0x7fcad4137a10 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad9a6c640 5 monclient: authenticate success, global_id 4237 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad9a6c640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad9a6c640 4 set_mon_vals no callback set 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fcad27fc640 10 monclient: dump: 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 10 monclient: shutdown 2024-04-23T16:39:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:44.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== osd.1 v2:172.21.15.79:6803/3615156451 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7aa40ace50 con 0x7f7aa003bf30 2024-04-23T16:39:44.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 1 -- 172.21.15.79:0/3187001137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2540a3600 msgr2=0x7fd2540a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 1 --2- 172.21.15.79:0/3187001137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2540a3600 0x7fd2540a39e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2540a7100 tx=0x7fd24801c3a0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 1 -- 172.21.15.79:0/3187001137 shutdown_connections 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 1 --2- 172.21.15.79:0/3187001137 >> v2:172.21.15.79:3300/0 conn(0x7fd254138090 0x7fd2540a4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 1 --2- 172.21.15.79:0/3187001137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2540a3600 0x7fd2540a39e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.929+0000 7fd25adc1640 1 -- 172.21.15.79:0/3187001137 >> 172.21.15.79:0/3187001137 conn(0x7fd2540aa3e0 msgr2=0x7fd2540a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 -- 172.21.15.79:0/3187001137 shutdown_connections 2024-04-23T16:39:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 -- 172.21.15.79:0/3187001137 wait complete. 2024-04-23T16:39:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient: shutdown 2024-04-23T16:39:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- 172.21.15.79:0/3349348064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad4137a10 msgr2=0x7fcad4137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 --2- 172.21.15.79:0/3349348064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad4137a10 0x7fcad4137df0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcad40753a0 tx=0x7fcac401c430 comp rx=0 tx=0).stop 2024-04-23T16:39:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- 172.21.15.79:0/3349348064 shutdown_connections 2024-04-23T16:39:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 --2- 172.21.15.79:0/3349348064 >> v2:172.21.15.79:3300/0 conn(0x7fcad40a2be0 0x7fcad413f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 --2- 172.21.15.79:0/3349348064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad4137a10 0x7fcad4137df0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- 172.21.15.79:0/3349348064 >> 172.21.15.79:0/3349348064 conn(0x7fcad40a35c0 msgr2=0x7fcad40a39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- 172.21.15.79:0/3349348064 shutdown_connections 2024-04-23T16:39:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- 172.21.15.79:0/3349348064 wait complete. 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient: monmap: 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.004 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da1cfa640 1 -- 172.21.15.79:0/1498705707 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c0734a0 msgr2=0x7f4d9c133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da1cfa640 1 --2- 172.21.15.79:0/1498705707 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c0734a0 0x7f4d9c133cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000c820 tx=0x7f4d9000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 Processor -- start 2024-04-23T16:39:45.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 -- start start 2024-04-23T16:39:45.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient: init 2024-04-23T16:39:45.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90015070 con 0x7f4d9c0734a0 2024-04-23T16:39:45.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: dump: 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d90007c40 con 0x7f4d9c0734a0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 5 monclient: authenticate success, global_id 4214 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 10 monclient: _renew_subs 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d9c11ebb0 con 0x7f4d9c0734a0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 10 monclient: _renew_subs 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9000ce10 con 0x7f4d9c0734a0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d9c11f450 con 0x7f4d9c0734a0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 10 monclient: dump: 2024-04-23T16:39:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.008 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient: monmap: 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 Processor -- start 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f4d90004020 con 0x7f4d9c0734a0 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- start start 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient: init 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad409fe10 0x7fcad409f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): picked mon.noname-b con 0x7fcad409fe10 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 0x7fcad413c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): picked mon.noname-c con 0x7fcad40a2be0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcad4137a10 0x7fcad413c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): picked mon.noname-a con 0x7fcad4137a10 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcad4142500 con 0x7fcad4137a10 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 0x7fd2541cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): picked mon.noname-a con 0x7fd2540a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd254137c90 0x7fd25413e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): picked mon.noname-b con 0x7fd254137c90 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fd254138090 0x7fd25413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): picked mon.noname-c con 0x7fd254138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd2541422f0 con 0x7fd2540a3600 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd254142170 con 0x7fd254137c90 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fd254142470 con 0x7fd254138090 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd25adc1640 10 monclient(hunting): authenticate will time out at 939.125855s 2024-04-23T16:39:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 0x7fd2541cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 0x7fd2541cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47828/0 (socket says 172.21.15.79:47828) 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 -- 172.21.15.79:0/2964274162 learned_addr learned my addr 172.21.15.79:0/2964274162 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): get_auth_request con 0x7fd2540a3600 auth_method 0 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 1 --2- 172.21.15.79:0/2964274162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd254137c90 0x7fd25413e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): get_auth_request con 0x7fd254137c90 auth_method 0 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient(hunting): handle_auth_done global_id 4242 payload 824 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 -- 172.21.15.79:0/2964274162 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fd254138090 msgr2=0x7fd25413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 --2- 172.21.15.79:0/2964274162 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fd254138090 0x7fd25413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 -- 172.21.15.79:0/2964274162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd254137c90 msgr2=0x7fd25413e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 --2- 172.21.15.79:0/2964274162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd254137c90 0x7fd25413e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 monclient: found mon.noname-a 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fd258b5d640 1 -- 172.21.15.79:0/2964274162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2541cfca0 con 0x7fd2540a3600 2024-04-23T16:39:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcad40a04f0 con 0x7fcad409fe10 2024-04-23T16:39:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcad4142680 con 0x7fcad40a2be0 2024-04-23T16:39:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad9a6c640 10 monclient(hunting): authenticate will time out at 939.126648s 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6812/722020384 conn(0x7f4d800363a0 0x7f4d80038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f4d90048d40 con 0x7f4d9c0734a0 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da0cf8640 4 set_mon_vals no callback set 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6808/3961528928 conn(0x7f4d9c11d730 0x7f4d9c11db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 --> v2:172.21.15.204:6808/3961528928 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d9c120320 con 0x7f4d9c11d730 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 0x7fcad413c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 0x7fcad413c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47180/0 (socket says 172.21.15.79:47180) 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 -- 172.21.15.79:0/1853826407 learned_addr learned my addr 172.21.15.79:0/1853826407 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): get_auth_request con 0x7fcad40a2be0 auth_method 0 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da24fb640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6812/722020384 conn(0x7f4d800363a0 0x7f4d80038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da24fb640 10 monclient: get_auth_request con 0x7f4d800363a0 auth_method 0 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad3fff640 1 --2- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcad4137a10 0x7fcad413c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad3fff640 10 monclient(hunting): get_auth_request con 0x7fcad4137a10 auth_method 0 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da2cfc640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6808/3961528928 conn(0x7f4d9c11d730 0x7f4d9c11db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-04-23T16:39:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 -- 172.21.15.79:0/1853826407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad409fe10 msgr2=0x7fcad409f840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T16:39:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 --2- 172.21.15.79:0/1853826407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcad409fe10 0x7fcad409f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7f4da2cfc640 10 monclient: get_auth_request con 0x7f4d9c11d730 auth_method 0 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 -- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcad4137a10 msgr2=0x7fcad413c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 --2- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcad4137a10 0x7fcad413c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 monclient: found mon.noname-c 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.933+0000 7fcad2ffd640 1 -- 172.21.15.79:0/1853826407 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad409c680 con 0x7fcad40a2be0 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd258b5d640 1 -- 172.21.15.79:0/2964274162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 msgr2=0x7fd2541cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd258b5d640 1 --2- 172.21.15.79:0/2964274162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 0x7fd2541cf5f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd2480ae040 tx=0x7fd24801abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2480ba020 con 0x7fd2540a3600 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: dump: 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2480b6630 con 0x7fd2540a3600 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2480b0040 con 0x7fd2540a3600 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: dump: 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.023 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 5 monclient: authenticate success, global_id 4242 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 10 monclient: _renew_subs 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd252ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25413b940 con 0x7fd2540a3600 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 10 monclient: _renew_subs 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25413b6c0 con 0x7fd2540a3600 2024-04-23T16:39:45.025 INFO:teuthology.orchestra.run.smithi079.stdout:51539607554 2024-04-23T16:39:45.025 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T16:39:45.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7f4da24fb640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6812/722020384 conn(0x7f4d800363a0 0x7f4d80038860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d98000c10 tx=0x7f4d980a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd254068ae0 con 0x7fd2540a3600 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad2ffd640 1 -- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 msgr2=0x7fcad413c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad2ffd640 1 --2- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 0x7fcad413c2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcac800b410 tx=0x7fcac800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8015020 con 0x7fcad40a2be0 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: dump: 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.027 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac8011a40 con 0x7fcad40a2be0 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8011e30 con 0x7fcad40a2be0 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcabbfff640 10 monclient: dump: 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 5 monclient: authenticate success, global_id 4217 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 10 monclient: _renew_subs 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad413dec0 con 0x7fcad40a2be0 2024-04-23T16:39:45.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad1ffb640 4 set_mon_vals no callback set 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 10 monclient: _renew_subs 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad409bde0 con 0x7fcad40a2be0 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7fd2480bd020 con 0x7fd2540a3600 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6812/722020384 conn(0x7fd2380363a0 0x7fd238038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fd2480e35a0 con 0x7fd2540a3600 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6800/3194378740 conn(0x7fd23803bf30 0x7fd23803e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 --> v2:172.21.15.204:6800/3194378740 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2480b8ec0 con 0x7fd23803bf30 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fd2480e3880 con 0x7fd2540a3600 2024-04-23T16:39:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fd2517fa640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-23T16:39:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.937+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcad4068ae0 con 0x7fcad40a2be0 2024-04-23T16:39:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fd253fff640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6812/722020384 conn(0x7fd2380363a0 0x7fd238038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fd253fff640 10 monclient: get_auth_request con 0x7fd2380363a0 auth_method 0 2024-04-23T16:39:45.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7fcac8018020 con 0x7fcad40a2be0 2024-04-23T16:39:45.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.204:6812/722020384 conn(0x7fcac00363d0 0x7fcac0038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fd25935e640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6800/3194378740 conn(0x7fd23803bf30 0x7fd23803e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fcac8048a90 con 0x7fcad40a2be0 2024-04-23T16:39:45.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fd25935e640 10 monclient: get_auth_request con 0x7fd23803bf30 auth_method 0 2024-04-23T16:39:45.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7f4da2cfc640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6808/3961528928 conn(0x7f4d9c11d730 0x7f4d9c11db10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.79:6800/3678369310 conn(0x7fcac003bf60 0x7fcac003e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 --> v2:172.21.15.79:6800/3678369310 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcac801deb0 con 0x7fcac003bf60 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fcac804d2a0 con 0x7fcad40a2be0 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcabbfff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcad3fff640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.79:6800/3678369310 conn(0x7fcac003bf60 0x7fcac003e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcad3fff640 10 monclient: get_auth_request con 0x7fcac003bf60 auth_method 0 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcad37fe640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.204:6812/722020384 conn(0x7fcac00363d0 0x7fcac0038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcad37fe640 10 monclient: get_auth_request con 0x7fcac00363d0 auth_method 0 2024-04-23T16:39:45.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== osd.5 v2:172.21.15.204:6808/3961528928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d9c120320 con 0x7f4d9c11d730 2024-04-23T16:39:45.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fd253fff640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6812/722020384 conn(0x7fd2380363a0 0x7fd238038860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd25413bea0 tx=0x7fd240007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.941+0000 7fcad37fe640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.204:6812/722020384 conn(0x7fcac00363d0 0x7fcac0038890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcac40ab000 tx=0x7fcac40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.945+0000 7fd25935e640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6800/3194378740 conn(0x7fd23803bf30 0x7fd23803e350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.945+0000 7fcad3fff640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.79:6800/3678369310 conn(0x7fcac003bf60 0x7fcac003e380 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.945+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== osd.2 v2:172.21.15.79:6800/3678369310 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcac801deb0 con 0x7fcac003bf60 2024-04-23T16:39:45.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.945+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== osd.3 v2:172.21.15.204:6800/3194378740 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2480b8ec0 con 0x7fd23803bf30 2024-04-23T16:39:45.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 --> v2:172.21.15.79:6803/3615156451 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ab4134e70 con 0x7f7aa003bf30 2024-04-23T16:39:45.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7a9bfff640 1 -- 172.21.15.79:0/2385506209 <== osd.1 v2:172.21.15.79:6803/3615156451 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7ab4134e70 con 0x7f7aa003bf30 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 >> v2:172.21.15.79:6803/3615156451 conn(0x7f7aa003bf30 msgr2=0x7f7aa003e350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.79:6803/3615156451 conn(0x7f7aa003bf30 0x7f7aa003e350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 >> v2:172.21.15.204:6812/722020384 conn(0x7f7aa00363a0 msgr2=0x7f7aa0038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 --2- 172.21.15.79:0/2385506209 >> v2:172.21.15.204:6812/722020384 conn(0x7f7aa00363a0 0x7f7aa0038860 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7ab413bb30 tx=0x7f7aa8008040 comp rx=0 tx=0).stop 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 10 monclient: shutdown 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 msgr2=0x7f7ab41cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 --2- 172.21.15.79:0/2385506209 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7ab40a3600 0x7f7ab41cf8f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7aa401cbe0 tx=0x7f7aa40190e0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab37fe640 1 -- 172.21.15.79:0/2385506209 reap_dead start 2024-04-23T16:39:45.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 shutdown_connections 2024-04-23T16:39:45.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 >> 172.21.15.79:0/2385506209 conn(0x7f7ab40aa3e0 msgr2=0x7f7ab40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 shutdown_connections 2024-04-23T16:39:45.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.949+0000 7f7ab9844640 1 -- 172.21.15.79:0/2385506209 wait complete. 2024-04-23T16:39:45.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 --> v2:172.21.15.79:6800/3678369310 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcad4071d80 con 0x7fcac003bf60 2024-04-23T16:39:45.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcabbfff640 1 -- 172.21.15.79:0/1853826407 <== osd.2 v2:172.21.15.79:6800/3678369310 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcad4071d80 con 0x7fcac003bf60 2024-04-23T16:39:45.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 >> v2:172.21.15.79:6800/3678369310 conn(0x7fcac003bf60 msgr2=0x7fcac003e380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.79:6800/3678369310 conn(0x7fcac003bf60 0x7fcac003e380 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 >> v2:172.21.15.204:6812/722020384 conn(0x7fcac00363d0 msgr2=0x7fcac0038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 --2- 172.21.15.79:0/1853826407 >> v2:172.21.15.204:6812/722020384 conn(0x7fcac00363d0 0x7fcac0038890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcac40ab000 tx=0x7fcac40a6000 comp rx=0 tx=0).stop 2024-04-23T16:39:45.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 10 monclient: shutdown 2024-04-23T16:39:45.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 msgr2=0x7fcad413c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 --2- 172.21.15.79:0/1853826407 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcad40a2be0 0x7fcad413c2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcac800b410 tx=0x7fcac800b8d0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad3fff640 1 -- 172.21.15.79:0/1853826407 reap_dead start 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 shutdown_connections 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 >> 172.21.15.79:0/1853826407 conn(0x7fcad40a35c0 msgr2=0x7fcad4071670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 shutdown_connections 2024-04-23T16:39:45.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.961+0000 7fcad9a6c640 1 -- 172.21.15.79:0/1853826407 wait complete. 2024-04-23T16:39:45.042 INFO:teuthology.orchestra.run.smithi079.stdout:51539607556 2024-04-23T16:39:45.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T16:39:45.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.965+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 --> v2:172.21.15.204:6808/3961528928 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d9c1cdc00 con 0x7f4d9c11d730 2024-04-23T16:39:45.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4d8effd640 1 -- 172.21.15.79:0/1498705707 <== osd.5 v2:172.21.15.204:6808/3961528928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4d9c1cdc00 con 0x7f4d9c11d730 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6808/3961528928 conn(0x7f4d9c11d730 msgr2=0x7f4d9c11db10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6808/3961528928 conn(0x7f4d9c11d730 0x7f4d9c11db10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6812/722020384 conn(0x7f4d800363a0 msgr2=0x7f4d80038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 --2- 172.21.15.79:0/1498705707 >> v2:172.21.15.204:6812/722020384 conn(0x7f4d800363a0 0x7f4d80038860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d98000c10 tx=0x7f4d980a7000 comp rx=0 tx=0).stop 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 10 monclient: shutdown 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c0734a0 msgr2=0x7f4d9c133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 --2- 172.21.15.79:0/1498705707 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4d9c0734a0 0x7f4d9c133cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000c820 tx=0x7f4d9000cce0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da2cfc640 1 -- 172.21.15.79:0/1498705707 reap_dead start 2024-04-23T16:39:45.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 shutdown_connections 2024-04-23T16:39:45.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 >> 172.21.15.79:0/1498705707 conn(0x7f4d9c0a36a0 msgr2=0x7f4d9c1c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 shutdown_connections 2024-04-23T16:39:45.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.969+0000 7f4da475f640 1 -- 172.21.15.79:0/1498705707 wait complete. 2024-04-23T16:39:45.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 --> v2:172.21.15.204:6800/3194378740 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2540a8ab0 con 0x7fd23803bf30 2024-04-23T16:39:45.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd2517fa640 1 -- 172.21.15.79:0/2964274162 <== osd.3 v2:172.21.15.204:6800/3194378740 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd2540a8ab0 con 0x7fd23803bf30 2024-04-23T16:39:45.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6800/3194378740 conn(0x7fd23803bf30 msgr2=0x7fd23803e350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6800/3194378740 conn(0x7fd23803bf30 0x7fd23803e350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6812/722020384 conn(0x7fd2380363a0 msgr2=0x7fd238038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 --2- 172.21.15.79:0/2964274162 >> v2:172.21.15.204:6812/722020384 conn(0x7fd2380363a0 0x7fd238038860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd25413bea0 tx=0x7fd240007e40 comp rx=0 tx=0).stop 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 10 monclient: shutdown 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 msgr2=0x7fd2541cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 --2- 172.21.15.79:0/2964274162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd2540a3600 0x7fd2541cf5f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd2480ae040 tx=0x7fd24801abe0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25935e640 1 -- 172.21.15.79:0/2964274162 reap_dead start 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 shutdown_connections 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 >> 172.21.15.79:0/2964274162 conn(0x7fd2540aa3e0 msgr2=0x7fd2540a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 shutdown_connections 2024-04-23T16:39:45.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:44.973+0000 7fd25adc1640 1 -- 172.21.15.79:0/2964274162 wait complete. 2024-04-23T16:39:45.048 INFO:teuthology.orchestra.run.smithi079.stdout:51539607555 2024-04-23T16:39:45.048 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T16:39:45.049 INFO:teuthology.orchestra.run.smithi079.stdout:47244640258 2024-04-23T16:39:45.049 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T16:39:45.050 INFO:teuthology.orchestra.run.smithi079.stdout:47244640258 2024-04-23T16:39:45.050 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient: get_monmap_and_config 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient: monmap: 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:45.183613+0000 2024-04-23T16:39:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:45.183613+0000 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 Processor -- start 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 -- start start 2024-04-23T16:39:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient: init 2024-04-23T16:39:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-23T16:39:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50240a3600 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502413c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): picked mon.noname-b con 0x7f50240a39e0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f502413ce70 0x7f502413f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f502413ce70 0x7f502413f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f502413ce70 0x7f502413f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47852/0 (socket says 172.21.15.79:47852) 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502413c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502413c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47186/0 (socket says 172.21.15.79:47186) 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5028eb5640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f50240a3600 0x7f5024137c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:51230/0 (socket says 172.21.15.79:51230) 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): picked mon.noname-a con 0x7f502413ce70 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 1 -- 172.21.15.79:0/4165752041 learned_addr learned my addr 172.21.15.79:0/4165752041 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50241383a0 con 0x7f50240a3600 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f50240a0f10 con 0x7f502413ce70 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f50240a5d40 con 0x7f50240a39e0 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f502b119640 10 monclient(hunting): authenticate will time out at 939.371688s 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): get_auth_request con 0x7f502413ce70 auth_method 0 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): get_auth_request con 0x7f50240a39e0 auth_method 0 2024-04-23T16:39:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f50296b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.177+0000 7f5023fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 1 -- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 msgr2=0x7f502413c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 1 --2- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502413c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 1 -- 172.21.15.79:0/4165752041 >> v1:172.21.15.79:6789/0 conn(0x7f50240a3600 legacy=0x7f5024137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 1 monclient: found mon.noname-a 2024-04-23T16:39:45.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f50296b6640 1 -- 172.21.15.79:0/4165752041 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50240a1600 con 0x7f502413ce70 2024-04-23T16:39:45.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient: get_monmap_and_config 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient: monmap: 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:45.185701+0000 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:45.185701+0000 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:45.191 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 Processor -- start 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 -- start start 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient: init 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-23T16:39:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37e409bfd0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f37e409ba00 0x7f37e4138830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f37e409ba00 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f37e4138d70 0x7f37e413b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f37e4138d70 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e40a1c10 con 0x7f37e409bfd0 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f37e409be30 con 0x7f37e4138d70 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f37e409d920 con 0x7f37e409ba00 2024-04-23T16:39:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e8fd0640 10 monclient(hunting): authenticate will time out at 939.375086s 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f37e409ba00 0x7f37e4138830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e259c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f37e409bfd0 0x7f37e409c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.79:52760/0 (socket says 172.21.15.79:52760) 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e259c640 1 -- 172.21.15.79:0/23791798 learned_addr learned my addr 172.21.15.79:0/23791798 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): get_auth_request con 0x7f37e409ba00 auth_method 0 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.181+0000 7f37e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 1 -- 172.21.15.79:0/23791798 >> v2:172.21.15.79:3300/0 conn(0x7f37e4138d70 msgr2=0x7f37e413b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 1 --2- 172.21.15.79:0/23791798 >> v2:172.21.15.79:3300/0 conn(0x7f37e4138d70 0x7f37e413b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 1 -- 172.21.15.79:0/23791798 >> v1:172.21.15.204:6789/0 conn(0x7f37e409bfd0 legacy=0x7f37e409c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 1 monclient: found mon.noname-c 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f37e1d9b640 1 -- 172.21.15.79:0/23791798 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e409dc00 con 0x7f37e409ba00 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50296b6640 1 -- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f502413ce70 msgr2=0x7f502413f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50296b6640 1 --2- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f502413ce70 0x7f502413f260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f501c009e50 tx=0x7f501c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442ee5ccb5275f26 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 1 -- 172.21.15.79:0/4165752041 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c0a5530 con 0x7f502413ce70 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: dump: 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.199 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 1 -- 172.21.15.79:0/4165752041 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501c0a56d0 con 0x7f502413ce70 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 5 monclient: authenticate success, global_id 4248 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 4 set_mon_vals no callback set 2024-04-23T16:39:45.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 1 -- 172.21.15.79:0/4165752041 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c0a5aa0 con 0x7f502413ce70 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f50237fe640 10 monclient: dump: 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 10 monclient: shutdown 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f502413ce70 msgr2=0x7f502413f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 --2- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f502413ce70 0x7f502413f260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f501c009e50 tx=0x7f501c01c8b0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 shutdown_connections 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 --2- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f502413ce70 0x7f502413f260 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 --2- 172.21.15.79:0/4165752041 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502413c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 >> 172.21.15.79:0/4165752041 conn(0x7f50240aa3e0 msgr2=0x7f50240a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 shutdown_connections 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 -- 172.21.15.79:0/4165752041 wait complete. 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 10 monclient: monmap: 2024-04-23T16:39:45.202 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 Processor -- start 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 1 -- start start 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.185+0000 7f502b119640 10 monclient: init 2024-04-23T16:39:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 0x7f502413b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): picked mon.noname-a con 0x7f50240a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502409fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): picked mon.noname-c con 0x7f50240a39e0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f502413ce70 0x7f50240a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): picked mon.noname-b con 0x7f502413ce70 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5024142140 con 0x7f50240a3600 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5024141fc0 con 0x7f502413ce70 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f50241422c0 con 0x7f50240a39e0 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient(hunting): authenticate will time out at 939.380412s 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 0x7f502413b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 0x7f502413b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47866/0 (socket says 172.21.15.79:47866) 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 -- 172.21.15.79:0/1841695518 learned_addr learned my addr 172.21.15.79:0/1841695518 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): get_auth_request con 0x7f50240a3600 auth_method 0 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502409fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): get_auth_request con 0x7f50240a39e0 auth_method 0 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50296b6640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f502413ce70 0x7f50240a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50296b6640 10 monclient(hunting): get_auth_request con 0x7f502413ce70 auth_method 0 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50296b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50296b6640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50296b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 -- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 msgr2=0x7f502409fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502409fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 -- 172.21.15.79:0/1841695518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f502413ce70 msgr2=0x7f50240a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f502413ce70 0x7f50240a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 monclient: found mon.noname-a 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 -- 172.21.15.79:0/1841695518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502409e6b0 con 0x7f50240a3600 2024-04-23T16:39:45.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5023fff640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502409fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:45.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e1d9b640 1 -- 172.21.15.79:0/23791798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409ba00 msgr2=0x7f37e4138830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e1d9b640 1 --2- 172.21.15.79:0/23791798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409ba00 0x7f37e4138830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f37e40a3860 tx=0x7f37d801d760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fb564bd7cfc7974 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 1 -- 172.21.15.79:0/23791798 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d80ad070 con 0x7f37e409ba00 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: dump: 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.209 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 1 -- 172.21.15.79:0/23791798 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d80a7720 con 0x7f37e409ba00 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 5 monclient: authenticate success, global_id 4249 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 4 set_mon_vals no callback set 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 1 -- 172.21.15.79:0/23791798 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d800b040 con 0x7f37e409ba00 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:45.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e159a640 10 monclient: dump: 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 10 monclient: shutdown 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 -- 172.21.15.79:0/23791798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409ba00 msgr2=0x7f37e4138830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/23791798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409ba00 0x7f37e4138830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f37e40a3860 tx=0x7f37d801d760 comp rx=0 tx=0).stop 2024-04-23T16:39:45.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 -- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 msgr2=0x7f502413b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5028eb5640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 0x7f502413b2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f501800b430 tx=0x7f501800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 -- 172.21.15.79:0/23791798 shutdown_connections 2024-04-23T16:39:45.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/23791798 >> v2:172.21.15.79:3300/0 conn(0x7f37e4138d70 0x7f37e413b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/23791798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409ba00 0x7f37e4138830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 -- 172.21.15.79:0/23791798 >> 172.21.15.79:0/23791798 conn(0x7f37e40a5fd0 msgr2=0x7f37e40a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 -- 172.21.15.79:0/23791798 shutdown_connections 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f37e8fd0640 1 -- 172.21.15.79:0/23791798 wait complete. 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient: monmap: 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.213 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.214 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.214 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018015020 con 0x7f50240a3600 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: dump: 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.215 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5018011a40 con 0x7f50240a3600 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018011e30 con 0x7f50240a3600 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f50217fa640 10 monclient: dump: 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.217 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 5 monclient: authenticate success, global_id 4251 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient: _renew_subs 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50240a40e0 con 0x7f50240a3600 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f5022ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient: _renew_subs 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.189+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50241cfce0 con 0x7f50240a3600 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f502b119640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f502b119640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f502b119640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5024068ae0 con 0x7f50240a3600 2024-04-23T16:39:45.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 Processor -- start 2024-04-23T16:39:45.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 -- start start 2024-04-23T16:39:45.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient: init 2024-04-23T16:39:45.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 0x7f37e4137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f37e409ba00 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409bfd0 0x7f37e41327e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f37e409bfd0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37e4138d70 0x7f37e4132d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f37e4138d70 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f37e413e250 con 0x7f37e4138d70 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f37e413e0d0 con 0x7f37e409bfd0 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f37e413e3d0 con 0x7f37e409ba00 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e8fd0640 10 monclient(hunting): authenticate will time out at 939.385791s 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f5018018020 con 0x7f50240a3600 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f50217fa640 1 --2- 172.21.15.79:0/1841695518 >> v2:172.21.15.204:6812/722020384 conn(0x7f50080363a0 0x7f5008038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f5018048850 con 0x7f50240a3600 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 0x7f37e4137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 0x7f37e4137fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47204/0 (socket says 172.21.15.79:47204) 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 -- 172.21.15.79:0/891984265 learned_addr learned my addr 172.21.15.79:0/891984265 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): get_auth_request con 0x7f37e409ba00 auth_method 0 2024-04-23T16:39:45.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f5023fff640 1 --2- 172.21.15.79:0/1841695518 >> v2:172.21.15.204:6812/722020384 conn(0x7f50080363a0 0x7f5008038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f5023fff640 10 monclient: get_auth_request con 0x7f50080363a0 auth_method 0 2024-04-23T16:39:45.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e1d9b640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409bfd0 0x7f37e41327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-04-23T16:39:45.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 -- 172.21.15.79:0/891984265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409bfd0 msgr2=0x7f37e41327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409bfd0 0x7f37e41327e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 -- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37e4138d70 msgr2=0x7f37e4132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37e4138d70 0x7f37e4132d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 monclient: found mon.noname-c 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.193+0000 7f37e259c640 1 -- 172.21.15.79:0/891984265 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e4135dd0 con 0x7f37e409ba00 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e259c640 1 -- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 msgr2=0x7f37e4137fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e259c640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 0x7f37e4137fb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37d400c900 tx=0x7f37d400cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4007d80 con 0x7f37e409ba00 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: dump: 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d4008040 con 0x7f37e409ba00 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 5 monclient: authenticate success, global_id 4226 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 monclient: _renew_subs 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e4134b80 con 0x7f37e409ba00 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d400f630 con 0x7f37e409ba00 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 10 monclient: dump: 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.226 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.227 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.227 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.227 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.227 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.227 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f5023fff640 1 --2- 172.21.15.79:0/1841695518 >> v2:172.21.15.204:6812/722020384 conn(0x7f50080363a0 0x7f5008038860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f502409d0f0 tx=0x7f5010009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 monclient: _renew_subs 2024-04-23T16:39:45.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e4133430 con 0x7f37e409ba00 2024-04-23T16:39:45.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e0d99640 4 set_mon_vals no callback set 2024-04-23T16:39:45.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c8005090 con 0x7f37e409ba00 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f37d400f850 con 0x7f37e409ba00 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.197+0000 7f37d2ffd640 1 --2- 172.21.15.79:0/891984265 >> v2:172.21.15.204:6812/722020384 conn(0x7f37c40363a0 0x7f37c4038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f37d4049350 con 0x7f37e409ba00 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f37e1d9b640 1 --2- 172.21.15.79:0/891984265 >> v2:172.21.15.204:6812/722020384 conn(0x7f37c40363a0 0x7f37c4038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f37e1d9b640 10 monclient: get_auth_request con 0x7f37c40363a0 auth_method 0 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f501801f070 con 0x7f50240a3600 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f50217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f50217fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:45.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.201+0000 7f37e1d9b640 1 --2- 172.21.15.79:0/891984265 >> v2:172.21.15.204:6812/722020384 conn(0x7f37c40363a0 0x7f37c4038860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37d80af000 tx=0x7f37d80b4040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.209+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f37d4014030 con 0x7f37e409ba00 2024-04-23T16:39:45.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.209+0000 7f37d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.209+0000 7f37d2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:45.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.221+0000 7f333757c640 10 monclient: get_monmap_and_config 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.221+0000 7f333757c640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.221+0000 7f333757c640 10 monclient: monmap: 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:45.228978+0000 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:45.228978+0000 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:45.232 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 Processor -- start 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 -- start start 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient: init 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 0x7f33300a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): picked mon.noname-b con 0x7f33300a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3330137c90 0x7f33300a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): picked mon.noname-c con 0x7f3330137c90 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 0x7f33300a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 0x7f33300a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 0x7f33300a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47218/0 (socket says 172.21.15.79:47218) 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 0x7f33300a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 0x7f33300a4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47888/0 (socket says 172.21.15.79:47888) 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): picked mon.noname-a con 0x7f3330138090 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 -- 172.21.15.79:0/38156763 learned_addr learned my addr 172.21.15.79:0/38156763 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f33300a0f10 con 0x7f3330138090 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f33300a3f20 con 0x7f3330137c90 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f33300a5d40 con 0x7f33300a3600 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f333757c640 10 monclient(hunting): authenticate will time out at 939.417117s 2024-04-23T16:39:45.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): get_auth_request con 0x7f33300a3600 auth_method 0 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 1 --2- 172.21.15.79:0/38156763 >> v2:172.21.15.204:3300/0 conn(0x7f3330137c90 0x7f33300a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): get_auth_request con 0x7f3330138090 auth_method 0 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): get_auth_request con 0x7f3330137c90 auth_method 0 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3334b17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 -- 172.21.15.79:0/38156763 >> v2:172.21.15.204:3300/0 conn(0x7f3330137c90 msgr2=0x7f33300a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 --2- 172.21.15.79:0/38156763 >> v2:172.21.15.204:3300/0 conn(0x7f3330137c90 0x7f33300a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 -- 172.21.15.79:0/38156763 >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 msgr2=0x7f33300a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 --2- 172.21.15.79:0/38156763 >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 0x7f33300a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 monclient: found mon.noname-b 2024-04-23T16:39:45.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335318640 1 -- 172.21.15.79:0/38156763 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33300a1600 con 0x7f33300a3600 2024-04-23T16:39:45.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.225+0000 7f3335b19640 1 --2- 172.21.15.79:0/38156763 >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 0x7f33300a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3335318640 1 -- 172.21.15.79:0/38156763 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 msgr2=0x7f33300a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3335318640 1 --2- 172.21.15.79:0/38156763 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 0x7f33300a39e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f333006d050 tx=0x7f332c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f8d6f0efeb82ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 1 -- 172.21.15.79:0/38156763 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0b0070 con 0x7f33300a3600 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: dump: 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.241 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 1 -- 172.21.15.79:0/38156763 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332c0ab600 con 0x7f33300a3600 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 1 -- 172.21.15.79:0/38156763 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0ab9d0 con 0x7f33300a3600 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f3327fff640 10 monclient: dump: 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f333757c640 5 monclient: authenticate success, global_id 4229 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f333757c640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.229+0000 7f333757c640 4 set_mon_vals no callback set 2024-04-23T16:39:45.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient: shutdown 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 msgr2=0x7f33300a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- 172.21.15.79:0/38156763 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 0x7f33300a39e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f333006d050 tx=0x7f332c01c3e0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 shutdown_connections 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- 172.21.15.79:0/38156763 >> v2:172.21.15.79:3300/0 conn(0x7f3330138090 0x7f33300a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- 172.21.15.79:0/38156763 >> v2:172.21.15.204:3300/0 conn(0x7f3330137c90 0x7f33300a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- 172.21.15.79:0/38156763 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f33300a3600 0x7f33300a39e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 >> 172.21.15.79:0/38156763 conn(0x7f33300aa3e0 msgr2=0x7f33300a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 shutdown_connections 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/38156763 wait complete. 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient: monmap: 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.244 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 Processor -- start 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- start start 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient: init 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:39:45.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 0x7f33301cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): picked mon.noname-a con 0x7f33300a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3330137c90 0x7f3330136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 0x7f33301cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 0x7f33301cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47900/0 (socket says 172.21.15.79:47900) 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 1 -- 172.21.15.79:0/1384013067 learned_addr learned my addr 172.21.15.79:0/1384013067 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): picked mon.noname-b con 0x7f3330137c90 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 0x7f333013c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): picked mon.noname-c con 0x7f3330138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3330141370 con 0x7f33300a3600 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f33300a5670 con 0x7f3330137c90 2024-04-23T16:39:45.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f33301414f0 con 0x7f3330138090 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f333757c640 10 monclient(hunting): authenticate will time out at 939.427618s 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): get_auth_request con 0x7f33300a3600 auth_method 0 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 0x7f333013c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3330137c90 0x7f3330136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): get_auth_request con 0x7f3330138090 auth_method 0 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): get_auth_request con 0x7f3330137c90 auth_method 0 2024-04-23T16:39:45.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335b19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3334b17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.233+0000 7f3335318640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 -- 172.21.15.79:0/1384013067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3330137c90 msgr2=0x7f3330136920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3330137c90 0x7f3330136920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 -- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 msgr2=0x7f33301cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 0x7f33301cf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 monclient: found mon.noname-c 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33301222c0 con 0x7f3330138090 2024-04-23T16:39:45.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335318640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 0x7f33301cf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient: get_monmap_and_config 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient: monmap: 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:45.241931+0000 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:45.241931+0000 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:45.250 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 Processor -- start 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 -- start start 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient: init 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f1128b51640 10 monclient: get_monmap_and_config 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f1128b51640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f1128b51640 10 monclient: monmap: 2024-04-23T16:39:45.251 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:45.243407+0000 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:45.243407+0000 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:45.252 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0c3413bcf0 0x7f0c3413c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): picked mon.noname-a con 0x7f0c3413bcf0 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0c3409e730 0x7f0c340a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): picked mon.noname-c con 0x7f0c3409e730 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c3409c830 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c3409c200 con 0x7f0c3409c830 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f0c3409bd90 con 0x7f0c3413bcf0 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0c3409d4b0 con 0x7f0c3409e730 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c3a854640 10 monclient(hunting): authenticate will time out at 939.430036s 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0c3413bcf0 0x7f0c3413c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0c3413bcf0 0x7f0c3413c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47906/0 (socket says 172.21.15.79:47906) 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 -- 172.21.15.79:0/1385157596 learned_addr learned my addr 172.21.15.79:0/1385157596 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): get_auth_request con 0x7f0c3413bcf0 auth_method 0 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 1 --2- 172.21.15.79:0/1385157596 >> v2:172.21.15.204:3300/0 conn(0x7f0c3409e730 0x7f0c340a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): get_auth_request con 0x7f0c3409e730 auth_method 0 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 -- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 msgr2=0x7f333013c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3335b19640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 0x7f333013c870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f332800efc0 tx=0x7f332800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2024-04-23T16:39:45.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 -- 172.21.15.79:0/1385157596 >> v2:172.21.15.204:3300/0 conn(0x7f0c3409e730 msgr2=0x7f0c340a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 --2- 172.21.15.79:0/1385157596 >> v2:172.21.15.204:3300/0 conn(0x7f0c3409e730 0x7f0c340a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328018030 con 0x7f3330138090 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: dump: 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.258 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33280039a0 con 0x7f3330138090 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328003d70 con 0x7f3330138090 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f3325ffb640 10 monclient: dump: 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.259 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 5 monclient: authenticate success, global_id 4232 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f33277fe640 4 set_mon_vals no callback set 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 monclient: _renew_subs 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33300a88a0 con 0x7f3330138090 2024-04-23T16:39:45.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 -- 172.21.15.79:0/1385157596 >> v1:172.21.15.79:6789/0 conn(0x7f0c3409c830 legacy=0x7f0c340a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 monclient: found mon.noname-a 2024-04-23T16:39:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f0c33fff640 1 -- 172.21.15.79:0/1385157596 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3409d790 con 0x7f0c3413bcf0 2024-04-23T16:39:45.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 monclient: _renew_subs 2024-04-23T16:39:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33301208b0 con 0x7f3330138090 2024-04-23T16:39:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.237+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3330068ae0 con 0x7f3330138090 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient: get_monmap_and_config 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient: monmap: 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:45.245800+0000 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:45.245800+0000 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.265 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 Processor -- start 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 -- start start 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient: init 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2ec0a3820 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:45.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7ff2ec0a2320 0x7ff2ec09e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec0a2320 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ff2ec138dc0 0x7ff2ec13b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec138dc0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7ff2ec0a2320 0x7ff2ec09e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7ff2ec0a2320 0x7ff2ec09e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47920/0 (socket says 172.21.15.79:47920) 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 -- 172.21.15.79:0/1356057211 learned_addr learned my addr 172.21.15.79:0/1356057211 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2ec0a3c00 con 0x7ff2ec0a3820 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 1 --2- 172.21.15.79:0/1356057211 >> v2:172.21.15.204:3300/0 conn(0x7ff2ec138dc0 0x7ff2ec13b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7ff2ec0a2a70 con 0x7ff2ec0a2320 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7ff2ec09f000 con 0x7ff2ec138dc0 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f35e3640 10 monclient(hunting): authenticate will time out at 939.434386s 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): get_auth_request con 0x7ff2ec0a2320 auth_method 0 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): get_auth_request con 0x7ff2ec138dc0 auth_method 0 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 -- 172.21.15.79:0/1356057211 >> v2:172.21.15.204:3300/0 conn(0x7ff2ec138dc0 msgr2=0x7ff2ec13b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 --2- 172.21.15.79:0/1356057211 >> v2:172.21.15.204:3300/0 conn(0x7ff2ec138dc0 0x7ff2ec13b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 -- 172.21.15.79:0/1356057211 >> v1:172.21.15.204:6789/0 conn(0x7ff2ec0a3820 legacy=0x7ff2ec0a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 monclient: found mon.noname-a 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.241+0000 7ff2f0b7e640 1 -- 172.21.15.79:0/1356057211 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec09cee0 con 0x7ff2ec0a2320 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c33fff640 1 -- 172.21.15.79:0/1385157596 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413bcf0 msgr2=0x7f0c3413c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c33fff640 1 --2- 172.21.15.79:0/1385157596 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413bcf0 0x7f0c3413c0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009870 tx=0x7f0c2401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f667379c0cfbbc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 1 -- 172.21.15.79:0/1385157596 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c240b0070 con 0x7f0c3413bcf0 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: dump: 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.270 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 1 -- 172.21.15.79:0/1385157596 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c240ab600 con 0x7f0c3413bcf0 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 1 -- 172.21.15.79:0/1385157596 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c240ab9d0 con 0x7f0c3413bcf0 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c337fe640 10 monclient: dump: 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.271 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 5 monclient: authenticate success, global_id 4260 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:45.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 4 set_mon_vals no callback set 2024-04-23T16:39:45.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f332801b070 con 0x7f3330138090 2024-04-23T16:39:45.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f3325ffb640 1 --2- 172.21.15.79:0/1384013067 >> v2:172.21.15.204:6812/722020384 conn(0x7f3314036350 0x7f3314038810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f332804a2d0 con 0x7f3330138090 2024-04-23T16:39:45.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f3335318640 1 --2- 172.21.15.79:0/1384013067 >> v2:172.21.15.204:6812/722020384 conn(0x7f3314036350 0x7f3314038810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 10 monclient: shutdown 2024-04-23T16:39:45.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 -- 172.21.15.79:0/1385157596 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413bcf0 msgr2=0x7f0c3413c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 --2- 172.21.15.79:0/1385157596 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413bcf0 0x7f0c3413c0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009870 tx=0x7f0c2401c3a0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f3335318640 10 monclient: get_auth_request con 0x7f3314036350 auth_method 0 2024-04-23T16:39:45.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 -- 172.21.15.79:0/1385157596 shutdown_connections 2024-04-23T16:39:45.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 --2- 172.21.15.79:0/1385157596 >> v2:172.21.15.204:3300/0 conn(0x7f0c3409e730 0x7f0c340a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 --2- 172.21.15.79:0/1385157596 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413bcf0 0x7f0c3413c0d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 -- 172.21.15.79:0/1385157596 >> 172.21.15.79:0/1385157596 conn(0x7f0c340a5fd0 msgr2=0x7f0c340a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 -- 172.21.15.79:0/1385157596 shutdown_connections 2024-04-23T16:39:45.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 -- 172.21.15.79:0/1385157596 wait complete. 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 Processor -- start 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 -- start start 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient: init 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1124072560 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f112406f4c0 0x7f1124072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): picked mon.noname-c con 0x7f112406f4c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f1124072e80 0x7f112413b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): picked mon.noname-a con 0x7f1124072e80 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1124070900 con 0x7f1124072560 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f1124071230 con 0x7f1124072e80 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f11240a3fa0 con 0x7f112406f4c0 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1128b51640 10 monclient(hunting): authenticate will time out at 939.437813s 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f1124072e80 0x7f112413b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f1124072e80 0x7f112413b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47930/0 (socket says 172.21.15.79:47930) 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 -- 172.21.15.79:0/2018543373 learned_addr learned my addr 172.21.15.79:0/2018543373 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): get_auth_request con 0x7f1124072e80 auth_method 0 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1121d9b640 1 --2- 172.21.15.79:0/2018543373 >> v2:172.21.15.204:3300/0 conn(0x7f112406f4c0 0x7f1124072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1121d9b640 10 monclient(hunting): get_auth_request con 0x7f112406f4c0 auth_method 0 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1121d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1121d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1121d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 -- 172.21.15.79:0/2018543373 >> v2:172.21.15.204:3300/0 conn(0x7f112406f4c0 msgr2=0x7f1124072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 --2- 172.21.15.79:0/2018543373 >> v2:172.21.15.204:3300/0 conn(0x7f112406f4c0 0x7f1124072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 -- 172.21.15.79:0/2018543373 >> v1:172.21.15.204:6789/0 conn(0x7f1124072560 legacy=0x7f1124071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 monclient: found mon.noname-a 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f1122d9d640 1 -- 172.21.15.79:0/2018543373 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112413ca90 con 0x7f1124072e80 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 10 monclient: monmap: 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.279 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.280 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.280 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.280 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.280 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.280 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2f0b7e640 1 -- 172.21.15.79:0/1356057211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a2320 msgr2=0x7ff2ec09e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2f0b7e640 1 --2- 172.21.15.79:0/1356057211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a2320 0x7ff2ec09e940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0a2770 tx=0x7ff2dc01d5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f76b757f83f2bd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 1 -- 172.21.15.79:0/1356057211 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc0ad070 con 0x7ff2ec0a2320 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: dump: 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.281 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 1 -- 172.21.15.79:0/1356057211 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2dc0a7610 con 0x7ff2ec0a2320 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 1 -- 172.21.15.79:0/1356057211 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc0a79e0 con 0x7ff2ec0a2320 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2e3fff640 10 monclient: dump: 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.282 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2f35e3640 5 monclient: authenticate success, global_id 4263 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2f35e3640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:45.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7ff2f35e3640 4 set_mon_vals no callback set 2024-04-23T16:39:45.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 Processor -- start 2024-04-23T16:39:45.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 1 -- start start 2024-04-23T16:39:45.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.245+0000 7f0c3a854640 10 monclient: init 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient: shutdown 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a2320 msgr2=0x7ff2ec09e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/1356057211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a2320 0x7ff2ec09e940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0a2770 tx=0x7ff2dc01d5d0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 shutdown_connections 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/1356057211 >> v2:172.21.15.204:3300/0 conn(0x7ff2ec138dc0 0x7ff2ec13b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/1356057211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a2320 0x7ff2ec09e940 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 >> 172.21.15.79:0/1356057211 conn(0x7ff2ec0a2d00 msgr2=0x7ff2ec0a3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 shutdown_connections 2024-04-23T16:39:45.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/1356057211 wait complete. 2024-04-23T16:39:45.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3328022070 con 0x7f3330138090 2024-04-23T16:39:45.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f3325ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f3325ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:45.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:45.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 0x7f0c341d38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): picked mon.noname-c con 0x7f0c3409c830 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f3335318640 1 --2- 172.21.15.79:0/1384013067 >> v2:172.21.15.204:6812/722020384 conn(0x7f3314036350 0x7f3314038810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f332c0097f0 tx=0x7f332c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c34142f60 0x7f0c341433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 0x7f0c341d38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): picked mon.noname-b con 0x7f0c34142f60 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 0x7f0c341d38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47236/0 (socket says 172.21.15.79:47236) 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 0x7f0c3413dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient: monmap: 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.289 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 0x7f0c3413dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 0x7f0c3413dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47938/0 (socket says 172.21.15.79:47938) 2024-04-23T16:39:45.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): picked mon.noname-a con 0x7f0c3413d870 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 -- 172.21.15.79:0/2229094820 learned_addr learned my addr 172.21.15.79:0/2229094820 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c34142f60 0x7f0c341433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0c340a2240 con 0x7f0c3413d870 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0c34146420 con 0x7f0c34142f60 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f0c3413fea0 con 0x7f0c3409c830 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c3a854640 10 monclient(hunting): authenticate will time out at 939.441621s 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): get_auth_request con 0x7f0c3409c830 auth_method 0 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): get_auth_request con 0x7f0c3413d870 auth_method 0 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 Processor -- start 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): get_auth_request con 0x7f0c34142f60 auth_method 0 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- start start 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient: init 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-04-23T16:39:45.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 -- 172.21.15.79:0/2229094820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c34142f60 msgr2=0x7f0c341433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c34142f60 0x7f0c341433c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 -- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 msgr2=0x7f0c3413dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 0x7f0c3413dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 monclient: found mon.noname-c 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c33fff640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3409cc80 con 0x7f0c3409c830 2024-04-23T16:39:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f0c38df1640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 0x7f0c3413dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff2ec0a2320 0x7ff2ec1327e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec0a2320 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a3820 0x7ff2ec070720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f1122d9d640 1 -- 172.21.15.79:0/2018543373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124072e80 msgr2=0x7f112413b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f1122d9d640 1 --2- 172.21.15.79:0/2018543373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124072e80 0x7f112413b540 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f112406d180 tx=0x7f111401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79cf8b57ea4e14a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec0a3820 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a3820 0x7ff2ec070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a3820 0x7ff2ec070720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47952/0 (socket says 172.21.15.79:47952) 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 1 -- 172.21.15.79:0/3316904064 learned_addr learned my addr 172.21.15.79:0/3316904064 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 0x7ff2ec070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec138dc0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec13db20 con 0x7ff2ec0a3820 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec13d9a0 con 0x7ff2ec138dc0 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7ff2ec13dca0 con 0x7ff2ec0a2320 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f35e3640 10 monclient(hunting): authenticate will time out at 939.443244s 2024-04-23T16:39:45.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 1 -- 172.21.15.79:0/2018543373 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11140a5840 con 0x7f1124072e80 2024-04-23T16:39:45.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): get_auth_request con 0x7ff2ec0a3820 auth_method 0 2024-04-23T16:39:45.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: dump: 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.300 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 1 -- 172.21.15.79:0/2018543373 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11140a59e0 con 0x7f1124072e80 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f1128b51640 5 monclient: authenticate success, global_id 4266 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f1128b51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 1 -- 172.21.15.79:0/2018543373 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11140a5db0 con 0x7f1124072e80 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f112159a640 10 monclient: dump: 2024-04-23T16:39:45.301 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f1128b51640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:45.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7f1128b51640 4 set_mon_vals no callback set 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 0x7ff2ec070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f0b7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): get_auth_request con 0x7ff2ec138dc0 auth_method 0 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 -- 172.21.15.79:0/3316904064 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff2ec0a2320 msgr2=0x7ff2ec1327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff2ec0a2320 0x7ff2ec1327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 -- 172.21.15.79:0/3316904064 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a3820 msgr2=0x7ff2ec070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a3820 0x7ff2ec070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 monclient: found mon.noname-b 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.249+0000 7ff2f1b80640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec1cbcc0 con 0x7ff2ec138dc0 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient: shutdown 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- 172.21.15.79:0/2018543373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124072e80 msgr2=0x7f112413b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 --2- 172.21.15.79:0/2018543373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124072e80 0x7f112413b540 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f112406d180 tx=0x7f111401c870 comp rx=0 tx=0).stop 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- 172.21.15.79:0/2018543373 shutdown_connections 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 --2- 172.21.15.79:0/2018543373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124072e80 0x7f112413b540 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 --2- 172.21.15.79:0/2018543373 >> v2:172.21.15.204:3300/0 conn(0x7f112406f4c0 0x7f1124072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- 172.21.15.79:0/2018543373 >> 172.21.15.79:0/2018543373 conn(0x7f11240a7130 msgr2=0x7f11240a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- 172.21.15.79:0/2018543373 shutdown_connections 2024-04-23T16:39:45.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c33fff640 1 -- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 msgr2=0x7f0c341d38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c33fff640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 0x7f0c341d38f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c24018fb0 tx=0x7f0c24019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c240b0050 con 0x7f0c3409c830 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: dump: 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c240ba070 con 0x7f0c3409c830 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c240b47c0 con 0x7f0c3409c830 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 10 monclient: dump: 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.308 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 5 monclient: authenticate success, global_id 4235 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 monclient: _renew_subs 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c34141800 con 0x7f0c3409c830 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- 172.21.15.79:0/2018543373 wait complete. 2024-04-23T16:39:45.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient: build_initial_monmap 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient: monmap: 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:45.310 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 monclient: _renew_subs 2024-04-23T16:39:45.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 Processor -- start 2024-04-23T16:39:45.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c34141520 con 0x7f0c3409c830 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- start start 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient: init 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 0x7f11241dbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): picked mon.noname-b con 0x7f112406f4c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1124072560 0x7f11241dc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): picked mon.noname-c con 0x7f1124072560 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124130b10 0x7f1124130fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): picked mon.noname-a con 0x7f1124130b10 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:45.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f112413c620 con 0x7f1124130b10 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f112413c4a0 con 0x7f112406f4c0 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f112413c7a0 con 0x7f1124072560 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1128b51640 10 monclient(hunting): authenticate will time out at 939.446204s 2024-04-23T16:39:45.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c32ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124130b10 0x7f1124130fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124130b10 0x7f1124130fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47962/0 (socket says 172.21.15.79:47962) 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 1 -- 172.21.15.79:0/829431851 learned_addr learned my addr 172.21.15.79:0/829431851 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): get_auth_request con 0x7f1124130b10 auth_method 0 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f1b80640 1 -- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 msgr2=0x7ff2ec070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f1b80640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 0x7ff2ec070c60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400efc0 tx=0x7ff2e400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 0x7f11241dbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): get_auth_request con 0x7f112406f4c0 auth_method 0 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1122d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4018020 con 0x7ff2ec138dc0 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: dump: 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.316 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1124072560 0x7f11241dc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 5 monclient: authenticate success, global_id 4273 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 10 monclient: _renew_subs 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ec11d190 con 0x7ff2ec138dc0 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e40146a0 con 0x7ff2ec138dc0 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 10 monclient: _renew_subs 2024-04-23T16:39:45.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ec0a56b0 con 0x7ff2ec138dc0 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4014a90 con 0x7ff2ec138dc0 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e1ffb640 10 monclient: dump: 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.319 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7ff2e37fe640 4 set_mon_vals no callback set 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): get_auth_request con 0x7f1124072560 auth_method 0 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:45.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f1121d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 -- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1124072560 msgr2=0x7f11241dc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1124072560 0x7f11241dc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 -- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124130b10 msgr2=0x7f1124130fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124130b10 0x7f1124130fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 monclient: found mon.noname-b 2024-04-23T16:39:45.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:45.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f112259c640 1 -- 172.21.15.79:0/829431851 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1124073270 con 0x7f112406f4c0 2024-04-23T16:39:45.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c18005090 con 0x7f0c3409c830 2024-04-23T16:39:45.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f0c240b49e0 con 0x7f0c3409c830 2024-04-23T16:39:45.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 1 --2- 172.21.15.79:0/2229094820 >> v2:172.21.15.204:6812/722020384 conn(0x7f0c140363a0 0x7f0c14038860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.253+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f0c240e2d40 con 0x7f0c3409c830 2024-04-23T16:39:45.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2f35e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2f35e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2f35e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc005090 con 0x7ff2ec138dc0 2024-04-23T16:39:45.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7f0c2bfff640 1 --2- 172.21.15.79:0/2229094820 >> v2:172.21.15.204:6812/722020384 conn(0x7f0c140363a0 0x7f0c14038860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7f0c2bfff640 10 monclient: get_auth_request con 0x7f0c140363a0 auth_method 0 2024-04-23T16:39:45.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7ff2e401b020 con 0x7ff2ec138dc0 2024-04-23T16:39:45.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2e1ffb640 1 --2- 172.21.15.79:0/3316904064 >> v2:172.21.15.204:6812/722020384 conn(0x7ff2d0036350 0x7ff2d0038810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7ff2e4049d80 con 0x7ff2ec138dc0 2024-04-23T16:39:45.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2f137f640 1 --2- 172.21.15.79:0/3316904064 >> v2:172.21.15.204:6812/722020384 conn(0x7ff2d0036350 0x7ff2d0038810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7ff2f137f640 10 monclient: get_auth_request con 0x7ff2d0036350 auth_method 0 2024-04-23T16:39:45.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.257+0000 7f0c2bfff640 1 --2- 172.21.15.79:0/2229094820 >> v2:172.21.15.204:6812/722020384 conn(0x7f0c140363a0 0x7f0c14038860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c3413f110 tx=0x7f0c1c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.261+0000 7ff2f137f640 1 --2- 172.21.15.79:0/3316904064 >> v2:172.21.15.204:6812/722020384 conn(0x7ff2d0036350 0x7ff2d0038810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8003cf0 tx=0x7ff2e80079c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.261+0000 7f112259c640 1 -- 172.21.15.79:0/829431851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 msgr2=0x7f11241dbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:45.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.261+0000 7f112259c640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 0x7f11241dbdf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f111800c410 tx=0x7f111800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.261+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0c3409cc80 con 0x7f0c3409c830 2024-04-23T16:39:45.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.261+0000 7f0c317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.261+0000 7f0c317fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:45.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11180119a0 con 0x7f112406f4c0 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: dump: 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1118017070 con 0x7f112406f4c0 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118011d80 con 0x7f112406f4c0 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1112ffd640 10 monclient: dump: 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.328 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1120d99640 4 set_mon_vals no callback set 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 5 monclient: authenticate success, global_id 4276 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 monclient: _renew_subs 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11241349d0 con 0x7f112406f4c0 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 monclient: _renew_subs 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11241346f0 con 0x7f112406f4c0 2024-04-23T16:39:45.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff2e4030070 con 0x7ff2ec138dc0 2024-04-23T16:39:45.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7ff2e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7ff2e1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.269+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1124068ae0 con 0x7f112406f4c0 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.273+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 4) ==== 44233+0+0 (secure 0 0 0) 0x7f111801c460 con 0x7f112406f4c0 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.273+0000 7f1112ffd640 1 --2- 172.21.15.79:0/829431851 >> v2:172.21.15.204:6812/722020384 conn(0x7f11040363d0 0x7f1104038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.273+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f11180496d0 con 0x7f112406f4c0 2024-04-23T16:39:45.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.273+0000 7f1121d9b640 1 --2- 172.21.15.79:0/829431851 >> v2:172.21.15.204:6812/722020384 conn(0x7f11040363d0 0x7f1104038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:45.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.273+0000 7f1121d9b640 10 monclient: get_auth_request con 0x7f11040363d0 auth_method 0 2024-04-23T16:39:45.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.277+0000 7f1121d9b640 1 --2- 172.21.15.79:0/829431851 >> v2:172.21.15.204:6812/722020384 conn(0x7f11040363d0 0x7f1104038890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11240a1830 tx=0x7f111c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:45.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.285+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1124073270 con 0x7f112406f4c0 2024-04-23T16:39:45.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.285+0000 7f1112ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:45.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.285+0000 7f1112ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:45.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-23T16:39:45.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-23T16:39:45.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f37c8005380 con 0x7f37e409ba00 2024-04-23T16:39:45.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37d2ffd640 1 -- 172.21.15.79:0/891984265 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f37d4016e10 con 0x7f37e409ba00 2024-04-23T16:39:45.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-23T16:39:45.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37d2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:45.409 INFO:teuthology.orchestra.run.smithi079.stdout:47244640257 2024-04-23T16:39:45.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 >> v2:172.21.15.204:6812/722020384 conn(0x7f37c40363a0 msgr2=0x7f37c4038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/891984265 >> v2:172.21.15.204:6812/722020384 conn(0x7f37c40363a0 0x7f37c4038860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37d80af000 tx=0x7f37d80b4040 comp rx=0 tx=0).stop 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 10 monclient: shutdown 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 msgr2=0x7f37e4137fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 0x7f37e4137fb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37d400c900 tx=0x7f37d400cdc0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 shutdown_connections 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/891984265 >> v2:172.21.15.204:6812/722020384 conn(0x7f37c40363a0 0x7f37c4038860 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37e4138d70 0x7f37e4132d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e409bfd0 0x7f37e41327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 --2- 172.21.15.79:0/891984265 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f37e409ba00 0x7f37e4137fb0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37d400c900 tx=0x7f37d400cdc0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 >> 172.21.15.79:0/891984265 conn(0x7f37e40a5fd0 msgr2=0x7f37e413a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 shutdown_connections 2024-04-23T16:39:45.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.405+0000 7f37e8fd0640 1 -- 172.21.15.79:0/891984265 wait complete. 2024-04-23T16:39:45.427 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.5 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0c18005380 con 0x7f0c3409c830 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c317fa640 1 -- 172.21.15.79:0/2229094820 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0c240160b0 con 0x7f0c3409c830 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-23T16:39:45.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c317fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:45.429 INFO:teuthology.orchestra.run.smithi079.stdout:51539607553 2024-04-23T16:39:45.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 >> v2:172.21.15.204:6812/722020384 conn(0x7f0c140363a0 msgr2=0x7f0c14038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 --2- 172.21.15.79:0/2229094820 >> v2:172.21.15.204:6812/722020384 conn(0x7f0c140363a0 0x7f0c14038860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c3413f110 tx=0x7f0c1c009190 comp rx=0 tx=0).stop 2024-04-23T16:39:45.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 10 monclient: shutdown 2024-04-23T16:39:45.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 msgr2=0x7f0c341d38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 0x7f0c341d38f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c24018fb0 tx=0x7f0c24019140 comp rx=0 tx=0).stop 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 shutdown_connections 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 --2- 172.21.15.79:0/2229094820 >> v2:172.21.15.204:6812/722020384 conn(0x7f0c140363a0 0x7f0c14038860 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c3413d870 0x7f0c3413dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c34142f60 0x7f0c341433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 --2- 172.21.15.79:0/2229094820 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0c3409c830 0x7f0c341d38f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 >> 172.21.15.79:0/2229094820 conn(0x7f0c340a5fd0 msgr2=0x7f0c340725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 shutdown_connections 2024-04-23T16:39:45.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.425+0000 7f0c3a854640 1 -- 172.21.15.79:0/2229094820 wait complete. 2024-04-23T16:39:45.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.0 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f33301d05f0 con 0x7f3330138090 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f3325ffb640 1 -- 172.21.15.79:0/1384013067 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3328010040 con 0x7f3330138090 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f3325ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f3325ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:45.505 INFO:teuthology.orchestra.run.smithi079.stdout:51539607554 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 >> v2:172.21.15.204:6812/722020384 conn(0x7f3314036350 msgr2=0x7f3314038810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> v2:172.21.15.204:6812/722020384 conn(0x7f3314036350 0x7f3314038810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f332c0097f0 tx=0x7f332c0a6000 comp rx=0 tx=0).stop 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 10 monclient: shutdown 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 msgr2=0x7f333013c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 0x7f333013c870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f332800efc0 tx=0x7f332800c490 comp rx=0 tx=0).stop 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 shutdown_connections 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> v2:172.21.15.204:6812/722020384 conn(0x7f3314036350 0x7f3314038810 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f3330138090 0x7f333013c870 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3330137c90 0x7f3330136920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 --2- 172.21.15.79:0/1384013067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33300a3600 0x7f33301cf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 >> 172.21.15.79:0/1384013067 conn(0x7f33300aa3e0 msgr2=0x7f33300a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 shutdown_connections 2024-04-23T16:39:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.501+0000 7f333757c640 1 -- 172.21.15.79:0/1384013067 wait complete. 2024-04-23T16:39:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f502b119640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-23T16:39:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f502b119640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-23T16:39:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f502b119640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f502409cc80 con 0x7f50240a3600 2024-04-23T16:39:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f50217fa640 1 -- 172.21.15.79:0/1841695518 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f501800f4a0 con 0x7f50240a3600 2024-04-23T16:39:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f50217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-23T16:39:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.505+0000 7f50217fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:45.511 INFO:teuthology.orchestra.run.smithi079.stdout:51539607555 2024-04-23T16:39:45.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 >> v2:172.21.15.204:6812/722020384 conn(0x7f50080363a0 msgr2=0x7f5008038860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 --2- 172.21.15.79:0/1841695518 >> v2:172.21.15.204:6812/722020384 conn(0x7f50080363a0 0x7f5008038860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f502409d0f0 tx=0x7f5010009190 comp rx=0 tx=0).stop 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 10 monclient: shutdown 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 msgr2=0x7f502413b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 0x7f502413b2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f501800b430 tx=0x7f501800b8f0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 shutdown_connections 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 --2- 172.21.15.79:0/1841695518 >> v2:172.21.15.204:6812/722020384 conn(0x7f50080363a0 0x7f5008038860 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f502413ce70 0x7f50240a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f50240a39e0 0x7f502409fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 --2- 172.21.15.79:0/1841695518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50240a3600 0x7f502413b2d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 >> 172.21.15.79:0/1841695518 conn(0x7f50240aa3e0 msgr2=0x7f50240a2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 shutdown_connections 2024-04-23T16:39:45.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.509+0000 7f502b119640 1 -- 172.21.15.79:0/1841695518 wait complete. 2024-04-23T16:39:45.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.2 2024-04-23T16:39:45.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff2cc005650 con 0x7ff2ec138dc0 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2e1ffb640 1 -- 172.21.15.79:0/3316904064 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff2ec1cbcc0 con 0x7ff2ec138dc0 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2e1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:45.525 INFO:teuthology.orchestra.run.smithi079.stdout:51539607553 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 >> v2:172.21.15.204:6812/722020384 conn(0x7ff2d0036350 msgr2=0x7ff2d0038810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> v2:172.21.15.204:6812/722020384 conn(0x7ff2d0036350 0x7ff2d0038810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8003cf0 tx=0x7ff2e80079c0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 10 monclient: shutdown 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 msgr2=0x7ff2ec070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 0x7ff2ec070c60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400efc0 tx=0x7ff2e400c490 comp rx=0 tx=0).stop 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 shutdown_connections 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> v2:172.21.15.204:6812/722020384 conn(0x7ff2d0036350 0x7ff2d0038810 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2ec138dc0 0x7ff2ec070c60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff2ec0a3820 0x7ff2ec070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 --2- 172.21.15.79:0/3316904064 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff2ec0a2320 0x7ff2ec1327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.521+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 >> 172.21.15.79:0/3316904064 conn(0x7ff2ec0a2d00 msgr2=0x7ff2ec0a0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.525+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 shutdown_connections 2024-04-23T16:39:45.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.525+0000 7ff2f35e3640 1 -- 172.21.15.79:0/3316904064 wait complete. 2024-04-23T16:39:45.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.1 2024-04-23T16:39:45.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.4 2024-04-23T16:39:45.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.553+0000 7f1128b51640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-23T16:39:45.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.553+0000 7f1128b51640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-23T16:39:45.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.553+0000 7f1128b51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.204:3300/0 2024-04-23T16:39:45.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.553+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1124132f20 con 0x7f112406f4c0 2024-04-23T16:39:45.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1112ffd640 1 -- 172.21.15.79:0/829431851 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f111801bcc0 con 0x7f112406f4c0 2024-04-23T16:39:45.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1112ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-23T16:39:45.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1112ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:45.561 INFO:teuthology.orchestra.run.smithi079.stdout:47244640257 2024-04-23T16:39:45.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 >> v2:172.21.15.204:6812/722020384 conn(0x7f11040363d0 msgr2=0x7f1104038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 --2- 172.21.15.79:0/829431851 >> v2:172.21.15.204:6812/722020384 conn(0x7f11040363d0 0x7f1104038890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11240a1830 tx=0x7f111c008040 comp rx=0 tx=0).stop 2024-04-23T16:39:45.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 10 monclient: shutdown 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 msgr2=0x7f11241dbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 0x7f11241dbdf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f111800c410 tx=0x7f111800c8d0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 shutdown_connections 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 --2- 172.21.15.79:0/829431851 >> v2:172.21.15.204:6812/722020384 conn(0x7f11040363d0 0x7f1104038890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124130b10 0x7f1124130fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1124072560 0x7f11241dc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 --2- 172.21.15.79:0/829431851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f112406f4c0 0x7f11241dbdf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:45.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 >> 172.21.15.79:0/829431851 conn(0x7f11240a7130 msgr2=0x7f1124075330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:45.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 shutdown_connections 2024-04-23T16:39:45.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:45.557+0000 7f1128b51640 1 -- 172.21.15.79:0/829431851 wait complete. 2024-04-23T16:39:45.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.3 2024-04-23T16:39:46.427 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T16:39:46.446 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T16:39:46.524 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T16:39:46.533 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T16:39:46.543 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T16:39:46.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.545+0000 7f8e64baa640 10 monclient: get_monmap_and_config 2024-04-23T16:39:46.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.545+0000 7f8e64baa640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.545+0000 7f8e64baa640 10 monclient: monmap: 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:46.553418+0000 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:46.553418+0000 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:46.552 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 Processor -- start 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 -- start start 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient: init 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f8e600a3600 0x7f8e600a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e600a3600 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8e60137c90 0x7f8e600a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e60137c90 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e60138090 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e600a5d40 con 0x7f8e60138090 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f8e600a0f10 con 0x7f8e600a3600 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8e600a3f20 con 0x7f8e60137c90 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e64baa640 10 monclient(hunting): authenticate will time out at 940.741103s 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f8e600a3600 0x7f8e600a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f8e600a3600 0x7f8e600a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47978/0 (socket says 172.21.15.79:47978) 2024-04-23T16:39:46.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 -- 172.21.15.79:0/3014111600 learned_addr learned my addr 172.21.15.79:0/3014111600 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): get_auth_request con 0x7f8e600a3600 auth_method 0 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 -- 172.21.15.79:0/3014111600 >> v2:172.21.15.204:3300/0 conn(0x7f8e60137c90 msgr2=0x7f8e600a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 --2- 172.21.15.79:0/3014111600 >> v2:172.21.15.204:3300/0 conn(0x7f8e60137c90 0x7f8e600a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 -- 172.21.15.79:0/3014111600 >> v1:172.21.15.204:6789/0 conn(0x7f8e60138090 legacy=0x7f8e600a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.549+0000 7f8e5e59c640 1 -- 172.21.15.79:0/3014111600 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e600a1600 con 0x7f8e600a3600 2024-04-23T16:39:46.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5e59c640 1 -- 172.21.15.79:0/3014111600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 msgr2=0x7f8e600a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5e59c640 1 --2- 172.21.15.79:0/3014111600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e600a39e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8e60137af0 tx=0x7f8e5001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09ec7af6e179ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 1 -- 172.21.15.79:0/3014111600 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500b0070 con 0x7f8e600a3600 2024-04-23T16:39:46.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: dump: 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 1 -- 172.21.15.79:0/3014111600 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e500ab600 con 0x7f8e600a3600 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 5 monclient: authenticate success, global_id 4278 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 1 -- 172.21.15.79:0/3014111600 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500ab9d0 con 0x7f8e600a3600 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 4 set_mon_vals no callback set 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e5d59a640 10 monclient: dump: 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.562 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 10 monclient: shutdown 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 -- 172.21.15.79:0/3014111600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 msgr2=0x7f8e600a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 --2- 172.21.15.79:0/3014111600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e600a39e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8e60137af0 tx=0x7f8e5001c3a0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 -- 172.21.15.79:0/3014111600 shutdown_connections 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 --2- 172.21.15.79:0/3014111600 >> v2:172.21.15.204:3300/0 conn(0x7f8e60137c90 0x7f8e600a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 --2- 172.21.15.79:0/3014111600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e600a39e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 -- 172.21.15.79:0/3014111600 >> 172.21.15.79:0/3014111600 conn(0x7f8e600aa3e0 msgr2=0x7f8e600a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 -- 172.21.15.79:0/3014111600 shutdown_connections 2024-04-23T16:39:46.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.553+0000 7f8e64baa640 1 -- 172.21.15.79:0/3014111600 wait complete. 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient: monmap: 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 Processor -- start 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 -- start start 2024-04-23T16:39:46.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient: init 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e601cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e600a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8e60137c90 0x7f8e6013e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e60137c90 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e60138090 0x7f8e6013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e60138090 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8e601422f0 con 0x7f8e600a3600 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8e60142170 con 0x7f8e60138090 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f8e60142470 con 0x7f8e60137c90 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e64baa640 10 monclient(hunting): authenticate will time out at 940.749726s 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e601cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e601cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47988/0 (socket says 172.21.15.79:47988) 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 -- 172.21.15.79:0/2825459882 learned_addr learned my addr 172.21.15.79:0/2825459882 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): get_auth_request con 0x7f8e600a3600 auth_method 0 2024-04-23T16:39:46.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8e60137c90 0x7f8e6013e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): get_auth_request con 0x7f8e60137c90 auth_method 0 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 -- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8e60137c90 msgr2=0x7f8e6013e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8e60137c90 0x7f8e6013e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 -- 172.21.15.79:0/2825459882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e60138090 msgr2=0x7f8e6013edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e60138090 0x7f8e6013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.557+0000 7f8e5e59c640 1 -- 172.21.15.79:0/2825459882 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6013bdd0 con 0x7f8e600a3600 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e5e59c640 1 -- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 msgr2=0x7f8e601cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e5e59c640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e601cf750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e500ae040 tx=0x7f8e5001abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500ba020 con 0x7f8e600a3600 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: dump: 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.571 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e500b6630 con 0x7f8e600a3600 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500b0040 con 0x7f8e600a3600 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 10 monclient: dump: 2024-04-23T16:39:46.572 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e5cd99640 4 set_mon_vals no callback set 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 5 monclient: authenticate success, global_id 4281 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 monclient: _renew_subs 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6013b050 con 0x7f8e600a3600 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 monclient: _renew_subs 2024-04-23T16:39:46.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e6013f2f0 con 0x7f8e600a3600 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f8e500bd020 con 0x7f8e600a3600 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e60068ae0 con 0x7f8e600a3600 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 1 --2- 172.21.15.79:0/2825459882 >> v2:172.21.15.204:6812/722020384 conn(0x7f8e40036430 0x7f8e400388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f8e500e3170 con 0x7f8e600a3600 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e5dd9b640 1 --2- 172.21.15.79:0/2825459882 >> v2:172.21.15.204:6812/722020384 conn(0x7f8e40036430 0x7f8e400388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.561+0000 7f8e5dd9b640 10 monclient: get_auth_request con 0x7f8e40036430 auth_method 0 2024-04-23T16:39:46.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.565+0000 7f8e5dd9b640 1 --2- 172.21.15.79:0/2825459882 >> v2:172.21.15.204:6812/722020384 conn(0x7f8e40036430 0x7f8e400388f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e6013d270 tx=0x7f8e54009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.573+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e500c4030 con 0x7f8e600a3600 2024-04-23T16:39:46.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.573+0000 7f8e4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.573+0000 7f8e4effd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:46.577 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient: get_monmap_and_config 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient: monmap: 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:46.603835+0000 2024-04-23T16:39:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:46.603835+0000 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 Processor -- start 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 -- start start 2024-04-23T16:39:46.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient: init 2024-04-23T16:39:46.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-23T16:39:46.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fc9240a3600 0x7fc9240a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): picked mon.noname-a con 0x7fc9240a3600 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc924137c90 0x7fc9240a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc924137c90 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc924138090 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9240a5d40 con 0x7fc924138090 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fc9240a0f10 con 0x7fc9240a3600 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc9240a3f20 con 0x7fc924137c90 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fc9240a3600 0x7fc9240a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fc9240a3600 0x7fc9240a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47992/0 (socket says 172.21.15.79:47992) 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 1 -- 172.21.15.79:0/1863253372 learned_addr learned my addr 172.21.15.79:0/1863253372 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc928b4b640 10 monclient(hunting): authenticate will time out at 940.791769s 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): get_auth_request con 0x7fc9240a3600 auth_method 0 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc92259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.597+0000 7fc921d9b640 1 --2- 172.21.15.79:0/1863253372 >> v2:172.21.15.204:3300/0 conn(0x7fc924137c90 0x7fc9240a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc921d9b640 10 monclient(hunting): get_auth_request con 0x7fc924137c90 auth_method 0 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc921d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc921d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 1 -- 172.21.15.79:0/1863253372 >> v2:172.21.15.204:3300/0 conn(0x7fc924137c90 msgr2=0x7fc9240a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 1 --2- 172.21.15.79:0/1863253372 >> v2:172.21.15.204:3300/0 conn(0x7fc924137c90 0x7fc9240a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 1 -- 172.21.15.79:0/1863253372 >> v1:172.21.15.204:6789/0 conn(0x7fc924138090 legacy=0x7fc9240a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.601+0000 7fc92259c640 1 -- 172.21.15.79:0/1863253372 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9240a1600 con 0x7fc9240a3600 2024-04-23T16:39:46.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 1 -- 172.21.15.79:0/1863253372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 msgr2=0x7fc9240a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 1 --2- 172.21.15.79:0/1863253372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9240a39e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc924137af0 tx=0x7fc91401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea39dc075087f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 1 -- 172.21.15.79:0/1863253372 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140b0070 con 0x7fc9240a3600 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: dump: 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.609 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 1 -- 172.21.15.79:0/1863253372 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9140ab600 con 0x7fc9240a3600 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 1 -- 172.21.15.79:0/1863253372 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140ab9d0 con 0x7fc9240a3600 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92159a640 10 monclient: dump: 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.610 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 5 monclient: authenticate success, global_id 4284 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:46.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 4 set_mon_vals no callback set 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: shutdown 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/1863253372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 msgr2=0x7fc9240a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 --2- 172.21.15.79:0/1863253372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9240a39e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc924137af0 tx=0x7fc91401c3c0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/1863253372 shutdown_connections 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 --2- 172.21.15.79:0/1863253372 >> v2:172.21.15.204:3300/0 conn(0x7fc924137c90 0x7fc9240a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 --2- 172.21.15.79:0/1863253372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9240a39e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/1863253372 >> 172.21.15.79:0/1863253372 conn(0x7fc9240aa3e0 msgr2=0x7fc9240a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/1863253372 shutdown_connections 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/1863253372 wait complete. 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: monmap: 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 Processor -- start 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- start start 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: init 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9241cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): picked mon.noname-a con 0x7fc9240a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc924137c90 0x7fc92413e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc924137c90 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc924138090 0x7fc92413edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9241cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): picked mon.noname-b con 0x7fc924138090 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9241cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48004/0 (socket says 172.21.15.79:48004) 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 1 -- 172.21.15.79:0/3560899227 learned_addr learned my addr 172.21.15.79:0/3560899227 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc9241422f0 con 0x7fc9240a3600 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc924142170 con 0x7fc924138090 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc924142470 con 0x7fc924137c90 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc928b4b640 10 monclient(hunting): authenticate will time out at 940.799823s 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): get_auth_request con 0x7fc9240a3600 auth_method 0 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc921d9b640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc924137c90 0x7fc92413e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc921d9b640 10 monclient(hunting): get_auth_request con 0x7fc924137c90 auth_method 0 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc921d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc921d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.605+0000 7fc92259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc921d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc921d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc921d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 -- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc924137c90 msgr2=0x7fc92413e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc924137c90 0x7fc92413e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 -- 172.21.15.79:0/3560899227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc924138090 msgr2=0x7fc92413edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc924138090 0x7fc92413edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9240a88f0 con 0x7fc9240a3600 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 -- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 msgr2=0x7fc9241cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc92259c640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9241cfa60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc9140ae040 tx=0x7fc914019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140bb070 con 0x7fc9240a3600 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: dump: 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9140aa5b0 con 0x7fc9240a3600 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc920d99640 4 set_mon_vals no callback set 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc928b4b640 5 monclient: authenticate success, global_id 4287 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140b0040 con 0x7fc9240a3600 2024-04-23T16:39:46.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc928b4b640 10 monclient: _renew_subs 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc928b4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc928b4b640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9240a40e0 con 0x7fc9240a3600 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.609+0000 7fc912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc912ffd640 10 monclient: dump: 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.621 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc928b4b640 10 monclient: _renew_subs 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc928b4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc928b4b640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92413d340 con 0x7fc9240a3600 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fc9140be070 con 0x7fc9240a3600 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc912ffd640 1 --2- 172.21.15.79:0/3560899227 >> v2:172.21.15.204:6812/722020384 conn(0x7fc9040363e0 0x7fc9040388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc910ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc910ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc910ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8fc005090 con 0x7fc9240a3600 2024-04-23T16:39:46.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.613+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fc9140e6120 con 0x7fc9240a3600 2024-04-23T16:39:46.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.621+0000 7fc921d9b640 1 --2- 172.21.15.79:0/3560899227 >> v2:172.21.15.204:6812/722020384 conn(0x7fc9040363e0 0x7fc9040388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.621+0000 7fc921d9b640 10 monclient: get_auth_request con 0x7fc9040363e0 auth_method 0 2024-04-23T16:39:46.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.621+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc9140160d0 con 0x7fc9240a3600 2024-04-23T16:39:46.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.621+0000 7fc912ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.621+0000 7fc912ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:46.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.621+0000 7fc921d9b640 1 --2- 172.21.15.79:0/3560899227 >> v2:172.21.15.204:6812/722020384 conn(0x7fc9040363e0 0x7fc9040388a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc92413b8d0 tx=0x7fc918008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient: get_monmap_and_config 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient: monmap: 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:46.647209+0000 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:46.647209+0000 2024-04-23T16:39:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:46.646 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 Processor -- start 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 -- start start 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient: init 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f73000a3600 0x7f73000a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f73000a3600 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7300137c90 0x7f73000a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f7300137c90 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:46.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7300138090 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73000a5d40 con 0x7f7300138090 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7300137c90 0x7f73000a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72fffff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f7300138090 0x7f73000a4d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:51248/0 (socket says 172.21.15.79:51248) 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72fffff640 1 -- 172.21.15.79:0/2882268429 learned_addr learned my addr 172.21.15.79:0/2882268429 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f73000a0f10 con 0x7f7300137c90 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f73000a3f20 con 0x7f73000a3600 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f7305ff4640 10 monclient(hunting): authenticate will time out at 940.835202s 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): get_auth_request con 0x7f7300137c90 auth_method 0 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 1 --2- 172.21.15.79:0/2882268429 >> v2:172.21.15.204:3300/0 conn(0x7f73000a3600 0x7f73000a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): get_auth_request con 0x7f73000a3600 auth_method 0 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 1 -- 172.21.15.79:0/2882268429 >> v2:172.21.15.204:3300/0 conn(0x7f73000a3600 msgr2=0x7f73000a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 1 --2- 172.21.15.79:0/2882268429 >> v2:172.21.15.204:3300/0 conn(0x7f73000a3600 0x7f73000a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 1 -- 172.21.15.79:0/2882268429 >> v1:172.21.15.79:6789/0 conn(0x7f7300138090 legacy=0x7f73000a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.641+0000 7f72feffd640 1 -- 172.21.15.79:0/2882268429 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73000a1600 con 0x7f7300137c90 2024-04-23T16:39:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72feffd640 1 -- 172.21.15.79:0/2882268429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7300137c90 msgr2=0x7f73000a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72feffd640 1 --2- 172.21.15.79:0/2882268429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7300137c90 0x7f73000a47e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f73000a7100 tx=0x7f72f401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4d5b7ee4312da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 1 -- 172.21.15.79:0/2882268429 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f40b0070 con 0x7f7300137c90 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: dump: 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 1 -- 172.21.15.79:0/2882268429 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f40ab600 con 0x7f7300137c90 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 5 monclient: authenticate success, global_id 4290 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 1 -- 172.21.15.79:0/2882268429 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f40ab9d0 con 0x7f7300137c90 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f72fe7fc640 10 monclient: dump: 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 4 set_mon_vals no callback set 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 10 monclient: shutdown 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7300137c90 msgr2=0x7f73000a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 --2- 172.21.15.79:0/2882268429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7300137c90 0x7f73000a47e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f73000a7100 tx=0x7f72f401c320 comp rx=0 tx=0).stop 2024-04-23T16:39:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 shutdown_connections 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 --2- 172.21.15.79:0/2882268429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7300137c90 0x7f73000a47e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 --2- 172.21.15.79:0/2882268429 >> v2:172.21.15.204:3300/0 conn(0x7f73000a3600 0x7f73000a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 >> 172.21.15.79:0/2882268429 conn(0x7f73000aa3e0 msgr2=0x7f73000a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 shutdown_connections 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 -- 172.21.15.79:0/2882268429 wait complete. 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 10 monclient: monmap: 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.659 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:46.659 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:46.659 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:46.659 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 Processor -- start 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 1 -- start start 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.649+0000 7f7305ff4640 10 monclient: init 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 0x7f73001367e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f73000a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7300137c90 0x7f730013c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 0x7f73001367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 0x7f73001367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48018/0 (socket says 172.21.15.79:48018) 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 -- 172.21.15.79:0/887952965 learned_addr learned my addr 172.21.15.79:0/887952965 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f7300137c90 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7300138090 0x7f730013c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f7300138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f73001422f0 con 0x7f73000a3600 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7300142170 con 0x7f7300137c90 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7300138090 0x7f730013c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7300142470 con 0x7f7300138090 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7300137c90 0x7f730013c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f7305ff4640 10 monclient(hunting): authenticate will time out at 940.844813s 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): get_auth_request con 0x7f73000a3600 auth_method 0 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): get_auth_request con 0x7f7300138090 auth_method 0 2024-04-23T16:39:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): get_auth_request con 0x7f7300137c90 auth_method 0 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 -- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7300138090 msgr2=0x7f730013c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7300138090 0x7f730013c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 -- 172.21.15.79:0/887952965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7300137c90 msgr2=0x7f730013c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7300137c90 0x7f730013c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73000a40e0 con 0x7f73000a3600 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 -- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 msgr2=0x7f73001367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72ff7fe640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 0x7f73001367e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f72f000e930 tx=0x7f72f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0017070 con 0x7f73000a3600 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72e7fff640 10 monclient: dump: 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.667 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.653+0000 7f72e7fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f000cc10 con 0x7f73000a3600 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 5 monclient: authenticate success, global_id 4293 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 10 monclient: _renew_subs 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73000a85b0 con 0x7f73000a3600 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0013470 con 0x7f73000a3600 2024-04-23T16:39:46.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 10 monclient: _renew_subs 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f7305ff4640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7300120eb0 con 0x7f73000a3600 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 10 monclient: dump: 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.669 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72fdffb640 4 set_mon_vals no callback set 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e5ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d8005090 con 0x7f73000a3600 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f72f001a070 con 0x7f73000a3600 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 1 --2- 172.21.15.79:0/887952965 >> v2:172.21.15.204:6812/722020384 conn(0x7f72ec0363e0 0x7f72ec0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f72f0049140 con 0x7f73000a3600 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72feffd640 1 --2- 172.21.15.79:0/887952965 >> v2:172.21.15.204:6812/722020384 conn(0x7f72ec0363e0 0x7f72ec0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.657+0000 7f72feffd640 10 monclient: get_auth_request con 0x7f72ec0363e0 auth_method 0 2024-04-23T16:39:46.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.661+0000 7f72feffd640 1 --2- 172.21.15.79:0/887952965 >> v2:172.21.15.204:6812/722020384 conn(0x7f72ec0363e0 0x7f72ec0388a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f72f40ab000 tx=0x7f72f40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.661+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f72f0013610 con 0x7f73000a3600 2024-04-23T16:39:46.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.661+0000 7f72e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.661+0000 7f72e7fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:46.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.701+0000 7fe52e43a640 10 monclient: get_monmap_and_config 2024-04-23T16:39:46.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.701+0000 7fe52e43a640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.701+0000 7fe52e43a640 10 monclient: monmap: 2024-04-23T16:39:46.706 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.706 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:46.708023+0000 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:46.708023+0000 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:46.707 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.701+0000 7fe52e43a640 1 Processor -- start 2024-04-23T16:39:46.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.701+0000 7fe52e43a640 1 -- start start 2024-04-23T16:39:46.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.701+0000 7fe52e43a640 10 monclient: init 2024-04-23T16:39:46.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fe5280a3600 0x7fe5280a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient(hunting): picked mon.noname-a con 0x7fe5280a3600 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe528137c90 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe528138090 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5280a5600 con 0x7fe528138090 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5280a5d40 con 0x7fe528137c90 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fe5280a0f10 con 0x7fe5280a3600 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe52e43a640 10 monclient(hunting): authenticate will time out at 940.896179s 2024-04-23T16:39:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fe5280a3600 0x7fe5280a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fe5280a3600 0x7fe5280a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48026/0 (socket says 172.21.15.79:48026) 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 -- 172.21.15.79:0/3523608181 learned_addr learned my addr 172.21.15.79:0/3523608181 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): get_auth_request con 0x7fe5280a3600 auth_method 0 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 -- 172.21.15.79:0/3523608181 >> v1:172.21.15.204:6789/0 conn(0x7fe528137c90 legacy=0x7fe5280a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 -- 172.21.15.79:0/3523608181 >> v1:172.21.15.79:6789/0 conn(0x7fe528138090 legacy=0x7fe5280a4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.705+0000 7fe527fff640 1 -- 172.21.15.79:0/3523608181 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5280a1600 con 0x7fe5280a3600 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe527fff640 1 -- 172.21.15.79:0/3523608181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5280a3600 msgr2=0x7fe5280a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe527fff640 1 --2- 172.21.15.79:0/3523608181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5280a3600 0x7fe5280a39e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe51801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e068b7b2744f510 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 1 -- 172.21.15.79:0/3523608181 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180b0070 con 0x7fe5280a3600 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: dump: 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe52e43a640 5 monclient: authenticate success, global_id 4296 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe52e43a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 1 -- 172.21.15.79:0/3523608181 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5180ab600 con 0x7fe5280a3600 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe52e43a640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe52e43a640 4 set_mon_vals no callback set 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 1 -- 172.21.15.79:0/3523608181 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180ab9d0 con 0x7fe5280a3600 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.709+0000 7fe526ffd640 10 monclient: dump: 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.718 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient: shutdown 2024-04-23T16:39:46.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- 172.21.15.79:0/3523608181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5280a3600 msgr2=0x7fe5280a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3523608181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5280a3600 0x7fe5280a39e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe51801c3a0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- 172.21.15.79:0/3523608181 shutdown_connections 2024-04-23T16:39:46.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3523608181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5280a3600 0x7fe5280a39e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- 172.21.15.79:0/3523608181 >> 172.21.15.79:0/3523608181 conn(0x7fe5280aa3e0 msgr2=0x7fe5280a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- 172.21.15.79:0/3523608181 shutdown_connections 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- 172.21.15.79:0/3523608181 wait complete. 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient: monmap: 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:46.720 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 Processor -- start 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- start start 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient: init 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 0x7fe528138800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): picked mon.noname-c con 0x7fe5280a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe528137c90 0x7fe528138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): picked mon.noname-b con 0x7fe528137c90 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe528138090 0x7fe528139280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): picked mon.noname-a con 0x7fe528138090 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe5281437a0 con 0x7fe528138090 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5280a3f20 con 0x7fe528137c90 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fe528143920 con 0x7fe5280a3600 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52e43a640 10 monclient(hunting): authenticate will time out at 940.906520s 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 0x7fe528138800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 0x7fe528138800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:47284/0 (socket says 172.21.15.79:47284) 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 -- 172.21.15.79:0/3970090348 learned_addr learned my addr 172.21.15.79:0/3970090348 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): get_auth_request con 0x7fe5280a3600 auth_method 0 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe528137c90 0x7fe528138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): get_auth_request con 0x7fe528137c90 auth_method 0 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe528138090 0x7fe528139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): get_auth_request con 0x7fe528138090 auth_method 0 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe5277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe52c9d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-04-23T16:39:46.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 -- 172.21.15.79:0/3970090348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe528137c90 msgr2=0x7fe528138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe528137c90 0x7fe528138d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 -- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe528138090 msgr2=0x7fe528139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe528138090 0x7fe528139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 monclient: found mon.noname-c 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.713+0000 7fe527fff640 1 -- 172.21.15.79:0/3970090348 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5280a8740 con 0x7fe5280a3600 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fc5ec86d640 10 monclient: get_monmap_and_config 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fc5ec86d640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fc5ec86d640 10 monclient: monmap: 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:46.723048+0000 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:46.723048+0000 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.726 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.727 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:46.727 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:46.727 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:46.727 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:46.727 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:46.727 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe527fff640 1 -- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 msgr2=0x7fe528138800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe527fff640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 0x7fe528138800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe518019fd0 tx=0x7fe51801abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180ba020 con 0x7fe5280a3600 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: dump: 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.728 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5180b6630 con 0x7fe5280a3600 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 5 monclient: authenticate success, global_id 4250 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 10 monclient: _renew_subs 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5280a40e0 con 0x7fe5280a3600 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180b0040 con 0x7fe5280a3600 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 10 monclient: dump: 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.730 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe525ffb640 4 set_mon_vals no callback set 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 10 monclient: _renew_subs 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52813c0c0 con 0x7fe5280a3600 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fe5180bd020 con 0x7fe5280a3600 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 1 --2- 172.21.15.79:0/3970090348 >> v2:172.21.15.204:6812/722020384 conn(0x7fe514036430 0x7fe5140388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fc5ec86d640 1 Processor -- start 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fc5ec86d640 1 -- start start 2024-04-23T16:39:46.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fc5ec86d640 10 monclient: init 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.717+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fe5180bea00 con 0x7fe5280a3600 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fe5277fe640 1 --2- 172.21.15.79:0/3970090348 >> v2:172.21.15.204:6812/722020384 conn(0x7fe514036430 0x7fe5140388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fe5277fe640 10 monclient: get_auth_request con 0x7fe514036430 auth_method 0 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fe52e43a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fe52e43a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fe52e43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-23T16:39:46.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe528068ae0 con 0x7fe5280a3600 2024-04-23T16:39:46.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027780 0x7fc5e00b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e0027780 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc5e0027b60 0x7fc5e0027f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e0027b60 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:46.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5e00bbdc0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:46.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e0024610 con 0x7fc5e00bbdc0 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc5e0024eb0 con 0x7fc5e0027b60 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e00248f0 con 0x7fc5e0027780 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5ec86d640 10 monclient(hunting): authenticate will time out at 940.912597s 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc5e0027b60 0x7fc5e0027f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e7fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc5e00bbdc0 0x7fc5e00bc220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.79:52834/0 (socket says 172.21.15.79:52834) 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e7fff640 1 -- 172.21.15.79:0/3056584318 learned_addr learned my addr 172.21.15.79:0/3056584318 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): get_auth_request con 0x7fc5e0027b60 auth_method 0 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/3056584318 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027780 msgr2=0x7fc5e00b8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 1 --2- 172.21.15.79:0/3056584318 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027780 0x7fc5e00b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/3056584318 >> v1:172.21.15.204:6789/0 conn(0x7fc5e00bbdc0 legacy=0x7fc5e00bc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 1 monclient: found mon.noname-c 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:46.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.721+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/3056584318 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e00b92e0 con 0x7fc5e0027b60 2024-04-23T16:39:46.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fe5277fe640 1 --2- 172.21.15.79:0/3970090348 >> v2:172.21.15.204:6812/722020384 conn(0x7fe514036430 0x7fe5140388f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe51c007fd0 tx=0x7fe51c007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/3056584318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027b60 msgr2=0x7fc5e0027f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e6ffd640 1 --2- 172.21.15.79:0/3056584318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027b60 0x7fc5e0027f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0097c0 tx=0x7fc5dc01c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=814fbc03ae4473b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 1 -- 172.21.15.79:0/3056584318 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0b0070 con 0x7fc5e0027b60 2024-04-23T16:39:46.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:46.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: dump: 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 1 -- 172.21.15.79:0/3056584318 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc0ab600 con 0x7fc5e0027b60 2024-04-23T16:39:46.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5ec86d640 5 monclient: authenticate success, global_id 4291 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 1 -- 172.21.15.79:0/3056584318 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0ab9d0 con 0x7fc5e0027b60 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5e67fc640 10 monclient: dump: 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5ec86d640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.725+0000 7fc5ec86d640 4 set_mon_vals no callback set 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient: shutdown 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- 172.21.15.79:0/3056584318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027b60 msgr2=0x7fc5e0027f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/3056584318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027b60 0x7fc5e0027f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0097c0 tx=0x7fc5dc01c320 comp rx=0 tx=0).stop 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- 172.21.15.79:0/3056584318 shutdown_connections 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/3056584318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027b60 0x7fc5e0027f40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/3056584318 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027780 0x7fc5e00b8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- 172.21.15.79:0/3056584318 >> 172.21.15.79:0/3056584318 conn(0x7fc5e0006800 msgr2=0x7fc5e0006c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- 172.21.15.79:0/3056584318 shutdown_connections 2024-04-23T16:39:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- 172.21.15.79:0/3056584318 wait complete. 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient: monmap: 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:46.743 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7f0ee7a22640 10 monclient: get_monmap_and_config 2024-04-23T16:39:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7f0ee7a22640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 Processor -- start 2024-04-23T16:39:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- start start 2024-04-23T16:39:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient: init 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7f0ee7a22640 10 monclient: monmap: 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:46.734709+0000 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:46.734709+0000 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:46.745 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027780 0x7fc5e0154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e0027780 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 0x7fc5e00c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e0027b60 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5e00bbdc0 0x7fc5e00c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e00bbdc0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e00c6cb0 con 0x7fc5e00bbdc0 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e00c6b30 con 0x7fc5e0027780 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e00c6e30 con 0x7fc5e0027b60 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5ec86d640 10 monclient(hunting): authenticate will time out at 940.921791s 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe5280a8740 con 0x7fe5280a3600 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fe524ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fe524ff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:46.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5e00bbdc0 0x7fc5e00c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5e00bbdc0 0x7fc5e00c33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48040/0 (socket says 172.21.15.79:48040) 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 1 -- 172.21.15.79:0/2329152320 learned_addr learned my addr 172.21.15.79:0/2329152320 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): get_auth_request con 0x7fc5e00bbdc0 auth_method 0 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 0x7fc5e00c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): get_auth_request con 0x7fc5e0027b60 auth_method 0 2024-04-23T16:39:46.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e77fe640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027780 0x7fc5e0154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e77fe640 10 monclient(hunting): get_auth_request con 0x7fc5e0027780 auth_method 0 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e77fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/2329152320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027780 msgr2=0x7fc5e0154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027780 0x7fc5e0154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5e00bbdc0 msgr2=0x7fc5e00c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5e00bbdc0 0x7fc5e00c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 monclient: found mon.noname-c 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.729+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/2329152320 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e00c1940 con 0x7fc5e0027b60 2024-04-23T16:39:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 Processor -- start 2024-04-23T16:39:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 -- start start 2024-04-23T16:39:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient: init 2024-04-23T16:39:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-23T16:39:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5e6ffd640 1 -- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 msgr2=0x7fc5e00c2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5e6ffd640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 0x7fc5e00c2e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0ab000 tx=0x7fc5dc01ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0ee00a3600 0x7f0ee00a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee00a3600 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ee0137c90 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0ee0138090 0x7f0ee00a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee0138090 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00a5d40 con 0x7f0ee0137c90 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00a0f10 con 0x7f0ee00a3600 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00a3f20 con 0x7f0ee0138090 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee7a22640 10 monclient(hunting): authenticate will time out at 940.924749s 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0b0050 con 0x7fc5e0027b60 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: dump: 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.754 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.755 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.757 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 5 monclient: authenticate success, global_id 4253 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 monclient: _renew_subs 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e002d5e0 con 0x7fc5e0027b60 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 monclient: _renew_subs 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e00c1660 con 0x7fc5e0027b60 2024-04-23T16:39:46.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0ee00a3600 0x7f0ee00a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0ee00a3600 0x7f0ee00a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48052/0 (socket says 172.21.15.79:48052) 2024-04-23T16:39:46.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 -- 172.21.15.79:0/198797578 learned_addr learned my addr 172.21.15.79:0/198797578 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): get_auth_request con 0x7f0ee00a3600 auth_method 0 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc0b1040 con 0x7fc5e0027b60 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0ab600 con 0x7fc5e0027b60 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:46.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5e5ffb640 4 set_mon_vals no callback set 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 10 monclient: dump: 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 -- 172.21.15.79:0/198797578 >> v2:172.21.15.204:3300/0 conn(0x7f0ee0138090 msgr2=0x7f0ee00a4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 --2- 172.21.15.79:0/198797578 >> v2:172.21.15.204:3300/0 conn(0x7f0ee0138090 0x7f0ee00a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 -- 172.21.15.79:0/198797578 >> v1:172.21.15.204:6789/0 conn(0x7f0ee0137c90 legacy=0x7f0ee00a47e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7f0ee57be640 1 -- 172.21.15.79:0/198797578 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee00a1600 con 0x7f0ee00a3600 2024-04-23T16:39:46.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c8005090 con 0x7fc5e0027b60 2024-04-23T16:39:46.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fc5dc0be020 con 0x7fc5e0027b60 2024-04-23T16:39:46.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 1 --2- 172.21.15.79:0/2329152320 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5cc0365b0 0x7fc5cc038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fc5dc0e3150 con 0x7fc5e0027b60 2024-04-23T16:39:46.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5e77fe640 1 --2- 172.21.15.79:0/2329152320 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5cc0365b0 0x7fc5cc038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.733+0000 7fc5e77fe640 10 monclient: get_auth_request con 0x7fc5cc0365b0 auth_method 0 2024-04-23T16:39:46.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7fc5e77fe640 1 --2- 172.21.15.79:0/2329152320 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5cc0365b0 0x7fc5cc038a70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4006fd0 tx=0x7fc5d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ee57be640 1 -- 172.21.15.79:0/198797578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee00a3600 msgr2=0x7f0ee00a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ee57be640 1 --2- 172.21.15.79:0/198797578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee00a3600 0x7f0ee00a39e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0ee006d230 tx=0x7f0ed801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd8760584f7a06a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 1 -- 172.21.15.79:0/198797578 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80a5640 con 0x7f0ee00a3600 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: dump: 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.765 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ee7a22640 5 monclient: authenticate success, global_id 4305 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ee7a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 1 -- 172.21.15.79:0/198797578 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed80a57e0 con 0x7f0ee00a3600 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ee7a22640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:46.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ee7a22640 4 set_mon_vals no callback set 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 1 -- 172.21.15.79:0/198797578 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80a5bb0 con 0x7f0ee00a3600 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.737+0000 7f0ed7fff640 10 monclient: dump: 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.767 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient: shutdown 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- 172.21.15.79:0/198797578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee00a3600 msgr2=0x7f0ee00a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/198797578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee00a3600 0x7f0ee00a39e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0ee006d230 tx=0x7f0ed801c870 comp rx=0 tx=0).stop 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- 172.21.15.79:0/198797578 shutdown_connections 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/198797578 >> v2:172.21.15.204:3300/0 conn(0x7f0ee0138090 0x7f0ee00a4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/198797578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee00a3600 0x7f0ee00a39e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- 172.21.15.79:0/198797578 >> 172.21.15.79:0/198797578 conn(0x7f0ee00aa3e0 msgr2=0x7f0ee00a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc5dc016030 con 0x7fc5e0027b60 2024-04-23T16:39:46.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7fc5d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7fc5d3fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- 172.21.15.79:0/198797578 shutdown_connections 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- 172.21.15.79:0/198797578 wait complete. 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient: build_initial_monmap 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient: monmap: 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:46.770 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 Processor -- start 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- start start 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient: init 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ee0137c90 0x7f0ee01cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee0137c90 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ee0138090 0x7f0ee013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee0138090 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:46.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 0x7f0ee0139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee013f040 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01422f0 con 0x7f0ee013f040 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0142170 con 0x7f0ee0138090 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee0142470 con 0x7f0ee0137c90 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee7a22640 10 monclient(hunting): authenticate will time out at 940.934788s 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 0x7f0ee0139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 0x7f0ee0139410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48058/0 (socket says 172.21.15.79:48058) 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 -- 172.21.15.79:0/2660059667 learned_addr learned my addr 172.21.15.79:0/2660059667 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): get_auth_request con 0x7f0ee013f040 auth_method 0 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:46.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 -- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ee0137c90 msgr2=0x7f0ee01cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee57be640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ee0137c90 0x7f0ee01cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ee0137c90 0x7f0ee01cf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 -- 172.21.15.79:0/2660059667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ee0138090 msgr2=0x7f0ee013eb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ee0138090 0x7f0ee013eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient: _finish_hunting 0 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 monclient: found mon.noname-a 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:46.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee5fbf640 1 -- 172.21.15.79:0/2660059667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee013d3a0 con 0x7f0ee013f040 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee57be640 10 monclient: get_auth_request con 0x7f0ee0137c90 auth_method 0 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.741+0000 7f0ee57be640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ee0137c90 0x7f0ee01cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee5fbf640 1 -- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 msgr2=0x7f0ee0139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee5fbf640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 0x7f0ee0139410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b430 tx=0x7f0ed000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0015020 con 0x7f0ee013f040 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: dump: 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.774 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed0011b20 con 0x7f0ee013f040 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0010380 con 0x7f0ee013f040 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 5 monclient: authenticate success, global_id 4308 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 monclient: _renew_subs 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee013b990 con 0x7f0ee013f040 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed77fe640 4 set_mon_vals no callback set 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 monclient: _renew_subs 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee013d0c0 con 0x7f0ee013f040 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 10 monclient: dump: 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee0068ae0 con 0x7f0ee013f040 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f0ed0018020 con 0x7f0ee013f040 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 1 --2- 172.21.15.79:0/2660059667 >> v2:172.21.15.204:6812/722020384 conn(0x7f0ec4036430 0x7f0ec40388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.745+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f0ed0049080 con 0x7f0ee013f040 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.753+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0ed001f020 con 0x7f0ee013f040 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.753+0000 7f0ed5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.753+0000 7f0ed5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.753+0000 7f0ee57be640 1 --2- 172.21.15.79:0/2660059667 >> v2:172.21.15.204:6812/722020384 conn(0x7f0ec4036430 0x7f0ec40388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.753+0000 7f0ee57be640 10 monclient: get_auth_request con 0x7f0ec4036430 auth_method 0 2024-04-23T16:39:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.753+0000 7f0ee57be640 1 --2- 172.21.15.79:0/2660059667 >> v2:172.21.15.204:6812/722020384 conn(0x7f0ec4036430 0x7f0ec40388f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ed80ab000 tx=0x7f0ed80a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:46.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc910ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-23T16:39:46.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc910ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-23T16:39:46.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc910ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc8fc005650 con 0x7fc9240a3600 2024-04-23T16:39:46.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc912ffd640 1 -- 172.21.15.79:0/3560899227 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc9140aab80 con 0x7fc9240a3600 2024-04-23T16:39:46.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc912ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-23T16:39:46.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.881+0000 7fc912ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:46.886 INFO:teuthology.orchestra.run.smithi079.stdout:51539607554 2024-04-23T16:39:46.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 >> v2:172.21.15.204:6812/722020384 conn(0x7fc9040363e0 msgr2=0x7fc9040388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 --2- 172.21.15.79:0/3560899227 >> v2:172.21.15.204:6812/722020384 conn(0x7fc9040363e0 0x7fc9040388a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc92413b8d0 tx=0x7fc918008040 comp rx=0 tx=0).stop 2024-04-23T16:39:46.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 10 monclient: shutdown 2024-04-23T16:39:46.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 msgr2=0x7fc9241cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9241cfa60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc9140ae040 tx=0x7fc914019160 comp rx=0 tx=0).stop 2024-04-23T16:39:46.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 shutdown_connections 2024-04-23T16:39:46.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 --2- 172.21.15.79:0/3560899227 >> v2:172.21.15.204:6812/722020384 conn(0x7fc9040363e0 0x7fc9040388a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc924138090 0x7fc92413edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc924137c90 0x7fc92413e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 --2- 172.21.15.79:0/3560899227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9240a3600 0x7fc9241cfa60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 >> 172.21.15.79:0/3560899227 conn(0x7fc9240aa3e0 msgr2=0x7fc92409ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 shutdown_connections 2024-04-23T16:39:46.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.885+0000 7fc910ff9640 1 -- 172.21.15.79:0/3560899227 wait complete. 2024-04-23T16:39:46.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e64baa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-23T16:39:46.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e64baa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-23T16:39:46.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e64baa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8e600a89e0 con 0x7f8e600a3600 2024-04-23T16:39:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e4effd640 1 -- 172.21.15.79:0/2825459882 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8e500160b0 con 0x7f8e600a3600 2024-04-23T16:39:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-23T16:39:46.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.889+0000 7f8e4effd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:46.895 INFO:teuthology.orchestra.run.smithi079.stdout:47244640258 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 >> v2:172.21.15.204:6812/722020384 conn(0x7f8e40036430 msgr2=0x7f8e400388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 --2- 172.21.15.79:0/2825459882 >> v2:172.21.15.204:6812/722020384 conn(0x7f8e40036430 0x7f8e400388f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e6013d270 tx=0x7f8e54009110 comp rx=0 tx=0).stop 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 10 monclient: shutdown 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 msgr2=0x7f8e601cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e601cf750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e500ae040 tx=0x7f8e5001abe0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 shutdown_connections 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 --2- 172.21.15.79:0/2825459882 >> v2:172.21.15.204:6812/722020384 conn(0x7f8e40036430 0x7f8e400388f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e60138090 0x7f8e6013edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8e60137c90 0x7f8e6013e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 --2- 172.21.15.79:0/2825459882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e600a3600 0x7f8e601cf750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 >> 172.21.15.79:0/2825459882 conn(0x7f8e600aa3e0 msgr2=0x7f8e600a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 shutdown_connections 2024-04-23T16:39:46.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.893+0000 7f8e64baa640 1 -- 172.21.15.79:0/2825459882 wait complete. 2024-04-23T16:39:46.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607554 for osd.0 2024-04-23T16:39:46.912 DEBUG:teuthology.parallel:result is None 2024-04-23T16:39:46.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.909+0000 7f0ee7a22640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-23T16:39:46.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.909+0000 7f0ee7a22640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-23T16:39:46.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.909+0000 7f0ee7a22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.909+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0ee01cff30 con 0x7f0ee013f040 2024-04-23T16:39:46.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ed5ffb640 1 -- 172.21.15.79:0/2660059667 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ed00107f0 con 0x7f0ee013f040 2024-04-23T16:39:46.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ed5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-23T16:39:46.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ed5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:46.916 INFO:teuthology.orchestra.run.smithi079.stdout:47244640258 2024-04-23T16:39:46.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5ec86d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-23T16:39:46.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5ec86d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-23T16:39:46.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5ec86d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc5c8005380 con 0x7fc5e0027b60 2024-04-23T16:39:46.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5d3fff640 1 -- 172.21.15.79:0/2329152320 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc5dc0b7450 con 0x7fc5e0027b60 2024-04-23T16:39:46.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-23T16:39:46.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7fc5d3fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:46.918 INFO:teuthology.orchestra.run.smithi079.stdout:51539607554 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 >> v2:172.21.15.204:6812/722020384 conn(0x7f0ec4036430 msgr2=0x7f0ec40388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/2660059667 >> v2:172.21.15.204:6812/722020384 conn(0x7f0ec4036430 0x7f0ec40388f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ed80ab000 tx=0x7f0ed80a7070 comp rx=0 tx=0).stop 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 10 monclient: shutdown 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 msgr2=0x7f0ee0139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 0x7f0ee0139410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b430 tx=0x7f0ed000b8f0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 shutdown_connections 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/2660059667 >> v2:172.21.15.204:6812/722020384 conn(0x7f0ec4036430 0x7f0ec40388f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ee013f040 0x7f0ee0139410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ee0138090 0x7f0ee013eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 --2- 172.21.15.79:0/2660059667 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0ee0137c90 0x7f0ee01cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 >> 172.21.15.79:0/2660059667 conn(0x7f0ee00aa3e0 msgr2=0x7f0ee009bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 shutdown_connections 2024-04-23T16:39:46.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.913+0000 7f0ee7a22640 1 -- 172.21.15.79:0/2660059667 wait complete. 2024-04-23T16:39:46.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5cc0365b0 msgr2=0x7fc5cc038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/2329152320 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5cc0365b0 0x7fc5cc038a70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4006fd0 tx=0x7fc5d4008040 comp rx=0 tx=0).stop 2024-04-23T16:39:46.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 10 monclient: shutdown 2024-04-23T16:39:46.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 msgr2=0x7fc5e00c2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 0x7fc5e00c2e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0ab000 tx=0x7fc5dc01ab60 comp rx=0 tx=0).stop 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 shutdown_connections 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/2329152320 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5cc0365b0 0x7fc5cc038a70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5e00bbdc0 0x7fc5e00c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5e0027b60 0x7fc5e00c2e70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 --2- 172.21.15.79:0/2329152320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0027780 0x7fc5e0154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 >> 172.21.15.79:0/2329152320 conn(0x7fc5e0006800 msgr2=0x7fc5e002bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 shutdown_connections 2024-04-23T16:39:46.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7fc5ec86d640 1 -- 172.21.15.79:0/2329152320 wait complete. 2024-04-23T16:39:46.923 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.5 2024-04-23T16:39:46.923 DEBUG:teuthology.parallel:result is None 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e5ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e5ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e5ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f72d8005380 con 0x7f73000a3600 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e7fff640 1 -- 172.21.15.79:0/887952965 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f72f001a520 con 0x7f73000a3600 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-23T16:39:46.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.917+0000 7f72e7fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:46.925 INFO:teuthology.orchestra.run.smithi079.stdout:51539607555 2024-04-23T16:39:46.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 >> v2:172.21.15.204:6812/722020384 conn(0x7f72ec0363e0 msgr2=0x7f72ec0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 --2- 172.21.15.79:0/887952965 >> v2:172.21.15.204:6812/722020384 conn(0x7f72ec0363e0 0x7f72ec0388a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f72f40ab000 tx=0x7f72f40a6000 comp rx=0 tx=0).stop 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 10 monclient: shutdown 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 msgr2=0x7f73001367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 0x7f73001367e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f72f000e930 tx=0x7f72f000edf0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 shutdown_connections 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 --2- 172.21.15.79:0/887952965 >> v2:172.21.15.204:6812/722020384 conn(0x7f72ec0363e0 0x7f72ec0388a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7300138090 0x7f730013c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7300137c90 0x7f730013c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 --2- 172.21.15.79:0/887952965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73000a3600 0x7f73001367e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 >> 172.21.15.79:0/887952965 conn(0x7f73000aa3e0 msgr2=0x7f73000a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 shutdown_connections 2024-04-23T16:39:46.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.921+0000 7f72e5ffb640 1 -- 172.21.15.79:0/887952965 wait complete. 2024-04-23T16:39:46.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.3 2024-04-23T16:39:46.939 DEBUG:teuthology.parallel:result is None 2024-04-23T16:39:46.940 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607554 for osd.4 2024-04-23T16:39:46.940 DEBUG:teuthology.parallel:result is None 2024-04-23T16:39:46.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607555 for osd.2 2024-04-23T16:39:46.944 DEBUG:teuthology.parallel:result is None 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.977+0000 7fe52e43a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.977+0000 7fe52e43a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.977+0000 7fe52e43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.977+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe528072040 con 0x7fe5280a3600 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe524ff9640 1 -- 172.21.15.79:0/3970090348 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe5180c4030 con 0x7fe5280a3600 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe524ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe524ff9640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:46.984 INFO:teuthology.orchestra.run.smithi079.stdout:51539607556 2024-04-23T16:39:46.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 >> v2:172.21.15.204:6812/722020384 conn(0x7fe514036430 msgr2=0x7fe5140388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3970090348 >> v2:172.21.15.204:6812/722020384 conn(0x7fe514036430 0x7fe5140388f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe51c007fd0 tx=0x7fe51c007e40 comp rx=0 tx=0).stop 2024-04-23T16:39:46.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 10 monclient: shutdown 2024-04-23T16:39:46.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 msgr2=0x7fe528138800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 0x7fe528138800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe518019fd0 tx=0x7fe51801abe0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 shutdown_connections 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3970090348 >> v2:172.21.15.204:6812/722020384 conn(0x7fe514036430 0x7fe5140388f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe528138090 0x7fe528139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe528137c90 0x7fe528138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 --2- 172.21.15.79:0/3970090348 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe5280a3600 0x7fe528138800 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 >> 172.21.15.79:0/3970090348 conn(0x7fe5280aa3e0 msgr2=0x7fe5280a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 shutdown_connections 2024-04-23T16:39:46.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:46.981+0000 7fe52e43a640 1 -- 172.21.15.79:0/3970090348 wait complete. 2024-04-23T16:39:47.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.1 2024-04-23T16:39:47.000 DEBUG:teuthology.parallel:result is None 2024-04-23T16:39:47.000 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T16:39:47.000 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient: get_monmap_and_config 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient: build_initial_monmap 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient: monmap: 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:47.147990+0000 2024-04-23T16:39:47.146 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:47.147990+0000 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:47.147 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 Processor -- start 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 -- start start 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient: init 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb54c0a3560 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb54c0a4c70 0x7fb54c13c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): picked mon.noname-a con 0x7fb54c0a4c70 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb54c0a5090 0x7fb54c13eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): picked mon.noname-c con 0x7fb54c0a5090 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:47.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb54c0a1560 con 0x7fb54c0a3560 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fb54c0a0f10 con 0x7fb54c0a4c70 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb54c0a1840 con 0x7fb54c0a5090 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb553426640 10 monclient(hunting): authenticate will time out at 941.335819s 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.141+0000 7fb5509c1640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb54c0a4c70 0x7fb54c13c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb54c0a4c70 0x7fb54c13c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35642/0 (socket says 172.21.15.79:35642) 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 -- 172.21.15.79:0/97732026 learned_addr learned my addr 172.21.15.79:0/97732026 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): get_auth_request con 0x7fb54c0a4c70 auth_method 0 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5519c3640 1 --2- 172.21.15.79:0/97732026 >> v2:172.21.15.204:3300/0 conn(0x7fb54c0a5090 0x7fb54c13eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5519c3640 10 monclient(hunting): get_auth_request con 0x7fb54c0a5090 auth_method 0 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5519c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5519c3640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5519c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 -- 172.21.15.79:0/97732026 >> v2:172.21.15.204:3300/0 conn(0x7fb54c0a5090 msgr2=0x7fb54c13eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:47.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 --2- 172.21.15.79:0/97732026 >> v2:172.21.15.204:3300/0 conn(0x7fb54c0a5090 0x7fb54c13eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 -- 172.21.15.79:0/97732026 >> v1:172.21.15.204:6789/0 conn(0x7fb54c0a3560 legacy=0x7fb54c0a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:47.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient: _finish_hunting 0 2024-04-23T16:39:47.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 monclient: found mon.noname-a 2024-04-23T16:39:47.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:47.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.145+0000 7fb5509c1640 1 -- 172.21.15.79:0/97732026 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c0a1b20 con 0x7fb54c0a4c70 2024-04-23T16:39:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5509c1640 1 -- 172.21.15.79:0/97732026 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a4c70 msgr2=0x7fb54c13c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5509c1640 1 --2- 172.21.15.79:0/97732026 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a4c70 0x7fb54c13c990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb54c076520 tx=0x7fb53c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fe1cbb06285430 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 1 -- 172.21.15.79:0/97732026 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c0b0070 con 0x7fb54c0a4c70 2024-04-23T16:39:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: dump: 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:47.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 5 monclient: authenticate success, global_id 4311 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 1 -- 172.21.15.79:0/97732026 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53c0ab600 con 0x7fb54c0a4c70 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 4 set_mon_vals no callback set 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 1 -- 172.21.15.79:0/97732026 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c0ab9d0 con 0x7fb54c0a4c70 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb543fff640 10 monclient: dump: 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.161 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: shutdown 2024-04-23T16:39:47.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- 172.21.15.79:0/97732026 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a4c70 msgr2=0x7fb54c13c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 --2- 172.21.15.79:0/97732026 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a4c70 0x7fb54c13c990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb54c076520 tx=0x7fb53c01c3a0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- 172.21.15.79:0/97732026 shutdown_connections 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 --2- 172.21.15.79:0/97732026 >> v2:172.21.15.204:3300/0 conn(0x7fb54c0a5090 0x7fb54c13eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 --2- 172.21.15.79:0/97732026 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a4c70 0x7fb54c13c990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- 172.21.15.79:0/97732026 >> 172.21.15.79:0/97732026 conn(0x7fb54c0aa490 msgr2=0x7fb54c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- 172.21.15.79:0/97732026 shutdown_connections 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- 172.21.15.79:0/97732026 wait complete. 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: build_initial_monmap 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: monmap: 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:47.163 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 Processor -- start 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- start start 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: init 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:47.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb54c0a3560 0x7fb54c1cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): picked mon.noname-c con 0x7fb54c0a3560 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb54c0a4c70 0x7fb54c136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): picked mon.noname-b con 0x7fb54c0a4c70 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 0x7fb54c13b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): picked mon.noname-a con 0x7fb54c0a5090 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c141fd0 con 0x7fb54c0a5090 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c141e50 con 0x7fb54c0a4c70 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fb54c142150 con 0x7fb54c0a3560 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb553426640 10 monclient(hunting): authenticate will time out at 941.343237s 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 0x7fb54c13b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb54c0a3560 0x7fb54c1cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 0x7fb54c13b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35646/0 (socket says 172.21.15.79:35646) 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 -- 172.21.15.79:0/416730943 learned_addr learned my addr 172.21.15.79:0/416730943 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): get_auth_request con 0x7fb54c0a5090 auth_method 0 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): get_auth_request con 0x7fb54c0a3560 auth_method 0 2024-04-23T16:39:47.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5511c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5509c1640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb54c0a4c70 0x7fb54c136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 -- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb54c0a3560 msgr2=0x7fb54c1cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb54c0a3560 0x7fb54c1cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 -- 172.21.15.79:0/416730943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb54c0a4c70 msgr2=0x7fb54c136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb54c0a4c70 0x7fb54c136340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient: _finish_hunting 0 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 monclient: found mon.noname-a 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5519c3640 1 -- 172.21.15.79:0/416730943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c120a10 con 0x7fb54c0a5090 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5509c1640 10 monclient: get_auth_request con 0x7fb54c0a4c70 auth_method 0 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.149+0000 7fb5509c1640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb54c0a4c70 0x7fb54c136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb5519c3640 1 -- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 msgr2=0x7fb54c13b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb5519c3640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 0x7fb54c13b640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb544007e40 tx=0x7fb54400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544013790 con 0x7fb54c0a5090 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: dump: 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 5 monclient: authenticate success, global_id 4314 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb544019070 con 0x7fb54c0a5090 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544013b90 con 0x7fb54c0a5090 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 10 monclient: dump: 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 monclient: _renew_subs 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c1d0170 con 0x7fb54c0a5090 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb542ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 monclient: _renew_subs 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c0a4060 con 0x7fb54c0a5090 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54c0692e0 con 0x7fb54c0a5090 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fb54401e460 con 0x7fb54c0a5090 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 1 --2- 172.21.15.79:0/416730943 >> v2:172.21.15.204:6812/722020384 conn(0x7fb530036460 0x7fb530038920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fb544049860 con 0x7fb54c0a5090 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb5511c2640 1 --2- 172.21.15.79:0/416730943 >> v2:172.21.15.204:6812/722020384 conn(0x7fb530036460 0x7fb530038920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.153+0000 7fb5511c2640 10 monclient: get_auth_request con 0x7fb530036460 auth_method 0 2024-04-23T16:39:47.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.157+0000 7fb5511c2640 1 --2- 172.21.15.79:0/416730943 >> v2:172.21.15.204:6812/722020384 conn(0x7fb530036460 0x7fb530038920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb548007fd0 tx=0x7fb548007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.157+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb544048750 con 0x7fb54c0a5090 2024-04-23T16:39:47.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.157+0000 7fb541ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.157+0000 7fb541ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:47.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.305+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb54c13a8c0 con 0x7fb530036460 2024-04-23T16:39:47.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb541ffb640 1 -- 172.21.15.79:0/416730943 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33856 (secure 0 0 0) 0x7fb54c13a8c0 con 0x7fb530036460 2024-04-23T16:39:47.312 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:47.313 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 >> v2:172.21.15.204:6812/722020384 conn(0x7fb530036460 msgr2=0x7fb530038920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 --2- 172.21.15.79:0/416730943 >> v2:172.21.15.204:6812/722020384 conn(0x7fb530036460 0x7fb530038920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb548007fd0 tx=0x7fb548007e40 comp rx=0 tx=0).stop 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 10 monclient: shutdown 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 msgr2=0x7fb54c13b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 0x7fb54c13b640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb544007e40 tx=0x7fb54400e4b0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 shutdown_connections 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 --2- 172.21.15.79:0/416730943 >> v2:172.21.15.204:6812/722020384 conn(0x7fb530036460 0x7fb530038920 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb54c0a5090 0x7fb54c13b640 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb54c0a4c70 0x7fb54c136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 --2- 172.21.15.79:0/416730943 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb54c0a3560 0x7fb54c1cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 >> 172.21.15.79:0/416730943 conn(0x7fb54c0aa490 msgr2=0x7fb54c1396b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 shutdown_connections 2024-04-23T16:39:47.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.309+0000 7fb553426640 1 -- 172.21.15.79:0/416730943 wait complete. 2024-04-23T16:39:47.329 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T16:39:47.298131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163004,"kb_used_data":848,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068036,"statfs":{"total":579820584960,"available":579653668864,"internally_reserved":0,"allocated":868352,"data_stored":1274966,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888464+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:39:44.888464+0000","last_peered":"2024-04-23T16:39:44.888464+0000","last_clean":"2024-04-23T16:39:44.888464+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:39:44.888464+0000","last_undegraded":"2024-04-23T16:39:44.888464+0000","last_fullsized":"2024-04-23T16:39:44.888464+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.705938+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:39:44.705938+0000","last_peered":"2024-04-23T16:39:44.705938+0000","last_clean":"2024-04-23T16:39:44.705938+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:39:44.705938+0000","last_undegraded":"2024-04-23T16:39:44.705938+0000","last_fullsized":"2024-04-23T16:39:44.705938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888524+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:39:44.888524+0000","last_peered":"2024-04-23T16:39:44.888524+0000","last_clean":"2024-04-23T16:39:44.888524+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:39:44.888524+0000","last_undegraded":"2024-04-23T16:39:44.888524+0000","last_fullsized":"2024-04-23T16:39:44.888524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.707572+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:39:44.707572+0000","last_peered":"2024-04-23T16:39:44.707572+0000","last_clean":"2024-04-23T16:39:44.707572+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:39:44.707572+0000","last_undegraded":"2024-04-23T16:39:44.707572+0000","last_fullsized":"2024-04-23T16:39:44.707572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888798+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:39:44.888798+0000","last_peered":"2024-04-23T16:39:44.888798+0000","last_clean":"2024-04-23T16:39:44.888798+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:39:44.888798+0000","last_undegraded":"2024-04-23T16:39:44.888798+0000","last_fullsized":"2024-04-23T16:39:44.888798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.887962+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:39:44.887962+0000","last_peered":"2024-04-23T16:39:44.887962+0000","last_clean":"2024-04-23T16:39:44.887962+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:39:44.887962+0000","last_undegraded":"2024-04-23T16:39:44.887962+0000","last_fullsized":"2024-04-23T16:39:44.887962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.706991+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:39:44.706991+0000","last_peered":"2024-04-23T16:39:44.706991+0000","last_clean":"2024-04-23T16:39:44.706991+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:39:44.706991+0000","last_undegraded":"2024-04-23T16:39:44.706991+0000","last_fullsized":"2024-04-23T16:39:44.706991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888472+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:39:44.888472+0000","last_peered":"2024-04-23T16:39:44.888472+0000","last_clean":"2024-04-23T16:39:44.888472+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:39:44.888472+0000","last_undegraded":"2024-04-23T16:39:44.888472+0000","last_fullsized":"2024-04-23T16:39:44.888472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888262+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:39:44.888262+0000","last_peered":"2024-04-23T16:39:44.888262+0000","last_clean":"2024-04-23T16:39:44.888262+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:39:44.888262+0000","last_undegraded":"2024-04-23T16:39:44.888262+0000","last_fullsized":"2024-04-23T16:39:44.888262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607554,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":176128,"data_stored":563091,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607554,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":34163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":567135,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:39:47.330 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:39:47.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient: get_monmap_and_config 2024-04-23T16:39:47.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient: build_initial_monmap 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient: monmap: 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:47.447789+0000 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:47.447789+0000 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:47.446 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 Processor -- start 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 -- start start 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient: init 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f5350135a20 0x7f5350135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5350135a20 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f535009f430 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5350135300 0x7f535009c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5350135300 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5350072f80 con 0x7f535009f430 2024-04-23T16:39:47.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f535009fb10 con 0x7f5350135a20 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f535009fdf0 con 0x7f5350135300 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f5355a1d640 10 monclient(hunting): authenticate will time out at 941.635497s 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534effd640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f5350135a20 0x7f5350135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534e7fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f535009f430 0x7f535009ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:46152/0 (socket says 172.21.15.79:46152) 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534e7fc640 1 -- 172.21.15.79:0/3816150823 learned_addr learned my addr 172.21.15.79:0/3816150823 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534effd640 10 monclient(hunting): get_auth_request con 0x7f5350135a20 auth_method 0 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534effd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.441+0000 7f534f7fe640 1 --2- 172.21.15.79:0/3816150823 >> v2:172.21.15.204:3300/0 conn(0x7f5350135300 0x7f535009c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): get_auth_request con 0x7f5350135300 auth_method 0 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 1 -- 172.21.15.79:0/3816150823 >> v2:172.21.15.79:3300/0 conn(0x7f5350135a20 msgr2=0x7f5350135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 1 --2- 172.21.15.79:0/3816150823 >> v2:172.21.15.79:3300/0 conn(0x7f5350135a20 0x7f5350135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 1 -- 172.21.15.79:0/3816150823 >> v1:172.21.15.79:6789/0 conn(0x7f535009f430 legacy=0x7f535009ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 1 monclient: found mon.noname-c 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:47.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.445+0000 7f534f7fe640 1 -- 172.21.15.79:0/3816150823 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535006de30 con 0x7f5350135300 2024-04-23T16:39:47.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 1 -- 172.21.15.79:0/3816150823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 msgr2=0x7f535009c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:47.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 1 --2- 172.21.15.79:0/3816150823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535009c110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5350136400 tx=0x7f534801c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e2ceb72159bdda server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 1 -- 172.21.15.79:0/3816150823 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480ac070 con 0x7f5350135300 2024-04-23T16:39:47.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: dump: 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 5 monclient: authenticate success, global_id 4300 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 1 -- 172.21.15.79:0/3816150823 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53480a6680 con 0x7f5350135300 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 1 -- 172.21.15.79:0/3816150823 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480a6a50 con 0x7f5350135300 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534dffb640 10 monclient: dump: 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.455 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 4 set_mon_vals no callback set 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: shutdown 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- 172.21.15.79:0/3816150823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 msgr2=0x7f535009c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3816150823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535009c110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5350136400 tx=0x7f534801c7b0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- 172.21.15.79:0/3816150823 shutdown_connections 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3816150823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535009c110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3816150823 >> v2:172.21.15.79:3300/0 conn(0x7f5350135a20 0x7f5350135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- 172.21.15.79:0/3816150823 >> 172.21.15.79:0/3816150823 conn(0x7f53500a25b0 msgr2=0x7f53500a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:47.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- 172.21.15.79:0/3816150823 shutdown_connections 2024-04-23T16:39:47.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- 172.21.15.79:0/3816150823 wait complete. 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: build_initial_monmap 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: monmap: 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:47.461 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 Processor -- start 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- start start 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: init 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 0x7f5350134000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): picked mon.noname-c con 0x7f535009f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535013bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5350135300 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 0x7f535013c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5350135a20 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f53501400b0 con 0x7f5350135a20 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f535009c860 con 0x7f5350135300 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5350140230 con 0x7f535009f430 2024-04-23T16:39:47.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f5355a1d640 10 monclient(hunting): authenticate will time out at 941.643533s 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 0x7f5350134000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 0x7f535013c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 0x7f5350134000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59882/0 (socket says 172.21.15.79:59882) 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 0x7f535013c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35674/0 (socket says 172.21.15.79:35674) 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 1 -- 172.21.15.79:0/3916315222 learned_addr learned my addr 172.21.15.79:0/3916315222 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): get_auth_request con 0x7f535009f430 auth_method 0 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): get_auth_request con 0x7f5350135a20 auth_method 0 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534e7fc640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535013bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534e7fc640 10 monclient(hunting): get_auth_request con 0x7f5350135300 auth_method 0 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.449+0000 7f534e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 -- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 msgr2=0x7f5350134000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 0x7f5350134000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 -- 172.21.15.79:0/3916315222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 msgr2=0x7f535013bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535013bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 monclient: found mon.noname-a 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 -- 172.21.15.79:0/3916315222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535011e990 con 0x7f5350135a20 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534e7fc640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535013bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534effd640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 0x7f5350134000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 -- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 msgr2=0x7f535013c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:47.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534f7fe640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 0x7f535013c430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f53480ab000 tx=0x7f5348019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480ac050 con 0x7f5350135a20 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: dump: 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.467 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 5 monclient: authenticate success, global_id 4320 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 monclient: _renew_subs 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5350072c70 con 0x7f5350135a20 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53480a6480 con 0x7f5350135a20 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480a7ae0 con 0x7f5350135a20 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 10 monclient: dump: 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.468 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534d7fa640 4 set_mon_vals no callback set 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 monclient: _renew_subs 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535011c6a0 con 0x7f5350135a20 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f53480c2060 con 0x7f5350135a20 2024-04-23T16:39:47.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53500692e0 con 0x7f5350135a20 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 1 --2- 172.21.15.79:0/3916315222 >> v2:172.21.15.204:6812/722020384 conn(0x7f53300363e0 0x7f53300388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f53480e31f0 con 0x7f5350135a20 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.453+0000 7f534effd640 1 --2- 172.21.15.79:0/3916315222 >> v2:172.21.15.204:6812/722020384 conn(0x7f53300363e0 0x7f53300388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.457+0000 7f534effd640 10 monclient: get_auth_request con 0x7f53300363e0 auth_method 0 2024-04-23T16:39:47.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.457+0000 7f534effd640 1 --2- 172.21.15.79:0/3916315222 >> v2:172.21.15.204:6812/722020384 conn(0x7f53300363e0 0x7f53300388a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5340007f00 tx=0x7f5340008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.461+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f53480164c0 con 0x7f5350135a20 2024-04-23T16:39:47.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.461+0000 7f5347fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.461+0000 7f5347fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:47.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.609+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f535011cf80 con 0x7f53300363e0 2024-04-23T16:39:47.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.609+0000 7f5347fff640 1 -- 172.21.15.79:0/3916315222 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33856 (secure 0 0 0) 0x7f535011cf80 con 0x7f53300363e0 2024-04-23T16:39:47.616 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:47.616 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:39:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 >> v2:172.21.15.204:6812/722020384 conn(0x7f53300363e0 msgr2=0x7f53300388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3916315222 >> v2:172.21.15.204:6812/722020384 conn(0x7f53300363e0 0x7f53300388a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5340007f00 tx=0x7f5340008040 comp rx=0 tx=0).stop 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 10 monclient: shutdown 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 msgr2=0x7f535013c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 0x7f535013c430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f53480ab000 tx=0x7f5348019550 comp rx=0 tx=0).stop 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 shutdown_connections 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3916315222 >> v2:172.21.15.204:6812/722020384 conn(0x7f53300363e0 0x7f53300388a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5350135a20 0x7f535013c430 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5350135300 0x7f535013bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 --2- 172.21.15.79:0/3916315222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f535009f430 0x7f5350134000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 >> 172.21.15.79:0/3916315222 conn(0x7f53500a25b0 msgr2=0x7f53500a39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 shutdown_connections 2024-04-23T16:39:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.613+0000 7f5355a1d640 1 -- 172.21.15.79:0/3916315222 wait complete. 2024-04-23T16:39:47.632 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T16:39:47.298131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163004,"kb_used_data":848,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068036,"statfs":{"total":579820584960,"available":579653668864,"internally_reserved":0,"allocated":868352,"data_stored":1274966,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888464+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:39:44.888464+0000","last_peered":"2024-04-23T16:39:44.888464+0000","last_clean":"2024-04-23T16:39:44.888464+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:39:44.888464+0000","last_undegraded":"2024-04-23T16:39:44.888464+0000","last_fullsized":"2024-04-23T16:39:44.888464+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.705938+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:39:44.705938+0000","last_peered":"2024-04-23T16:39:44.705938+0000","last_clean":"2024-04-23T16:39:44.705938+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:39:44.705938+0000","last_undegraded":"2024-04-23T16:39:44.705938+0000","last_fullsized":"2024-04-23T16:39:44.705938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888524+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:39:44.888524+0000","last_peered":"2024-04-23T16:39:44.888524+0000","last_clean":"2024-04-23T16:39:44.888524+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:39:44.888524+0000","last_undegraded":"2024-04-23T16:39:44.888524+0000","last_fullsized":"2024-04-23T16:39:44.888524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.707572+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:39:44.707572+0000","last_peered":"2024-04-23T16:39:44.707572+0000","last_clean":"2024-04-23T16:39:44.707572+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:39:44.707572+0000","last_undegraded":"2024-04-23T16:39:44.707572+0000","last_fullsized":"2024-04-23T16:39:44.707572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888798+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:39:44.888798+0000","last_peered":"2024-04-23T16:39:44.888798+0000","last_clean":"2024-04-23T16:39:44.888798+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:39:44.888798+0000","last_undegraded":"2024-04-23T16:39:44.888798+0000","last_fullsized":"2024-04-23T16:39:44.888798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.887962+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:39:44.887962+0000","last_peered":"2024-04-23T16:39:44.887962+0000","last_clean":"2024-04-23T16:39:44.887962+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:39:44.887962+0000","last_undegraded":"2024-04-23T16:39:44.887962+0000","last_fullsized":"2024-04-23T16:39:44.887962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.706991+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:39:44.706991+0000","last_peered":"2024-04-23T16:39:44.706991+0000","last_clean":"2024-04-23T16:39:44.706991+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:39:44.706991+0000","last_undegraded":"2024-04-23T16:39:44.706991+0000","last_fullsized":"2024-04-23T16:39:44.706991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888472+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:39:44.888472+0000","last_peered":"2024-04-23T16:39:44.888472+0000","last_clean":"2024-04-23T16:39:44.888472+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:39:44.888472+0000","last_undegraded":"2024-04-23T16:39:44.888472+0000","last_fullsized":"2024-04-23T16:39:44.888472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888262+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:39:44.888262+0000","last_peered":"2024-04-23T16:39:44.888262+0000","last_clean":"2024-04-23T16:39:44.888262+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:39:44.888262+0000","last_undegraded":"2024-04-23T16:39:44.888262+0000","last_fullsized":"2024-04-23T16:39:44.888262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607554,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":176128,"data_stored":563091,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607554,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":34163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":567135,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:39:47.633 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T16:39:47.633 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T16:39:47.633 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-23T16:39:47.633 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-23T16:39:47.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient: get_monmap_and_config 2024-04-23T16:39:47.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient: build_initial_monmap 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient: monmap: 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:47.750597+0000 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:47.750597+0000 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:47.749 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 Processor -- start 2024-04-23T16:39:47.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 -- start start 2024-04-23T16:39:47.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient: init 2024-04-23T16:39:47.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:47.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f8f8406ca30 0x7f8f840a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): picked mon.noname-a con 0x7f8f8406ca30 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 0x7f8f840a3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): picked mon.noname-b con 0x7f8f840a3370 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f840a47e0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f840a2f20 con 0x7f8f840a47e0 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f8f840a1b50 con 0x7f8f8406ca30 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f8f8406ce10 con 0x7f8f840a3370 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f8b369640 10 monclient(hunting): authenticate will time out at 941.938481s 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 0x7f8f840a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f8f8406ca30 0x7f8f840a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 0x7f8f840a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59894/0 (socket says 172.21.15.79:59894) 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 -- 172.21.15.79:0/3868623024 learned_addr learned my addr 172.21.15.79:0/3868623024 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): get_auth_request con 0x7f8f840a3370 auth_method 0 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): get_auth_request con 0x7f8f8406ca30 auth_method 0 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 -- 172.21.15.79:0/3868623024 >> v2:172.21.15.79:3300/0 conn(0x7f8f8406ca30 msgr2=0x7f8f840a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 --2- 172.21.15.79:0/3868623024 >> v2:172.21.15.79:3300/0 conn(0x7f8f8406ca30 0x7f8f840a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 -- 172.21.15.79:0/3868623024 >> v1:172.21.15.204:6789/0 conn(0x7f8f840a47e0 legacy=0x7f8f840a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient: _finish_hunting 0 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 monclient: found mon.noname-b 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f88904640 1 -- 172.21.15.79:0/3868623024 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f840a5860 con 0x7f8f840a3370 2024-04-23T16:39:47.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.745+0000 7f8f89105640 1 --2- 172.21.15.79:0/3868623024 >> v2:172.21.15.79:3300/0 conn(0x7f8f8406ca30 0x7f8f840a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f88904640 1 -- 172.21.15.79:0/3868623024 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 msgr2=0x7f8f840a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f88904640 1 --2- 172.21.15.79:0/3868623024 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 0x7f8f840a3750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009870 tx=0x7f8f7401c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f543fc32afeeea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 1 -- 172.21.15.79:0/3868623024 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f740b0070 con 0x7f8f840a3370 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: dump: 2024-04-23T16:39:47.758 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: _finish_auth 0 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:47.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 5 monclient: authenticate success, global_id 4262 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 1 -- 172.21.15.79:0/3868623024 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f740ab600 con 0x7f8f840a3370 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 4 set_mon_vals no callback set 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 1 -- 172.21.15.79:0/3868623024 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f740ab9d0 con 0x7f8f840a3370 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f7bfff640 10 monclient: dump: 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.760 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 10 monclient: shutdown 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 -- 172.21.15.79:0/3868623024 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 msgr2=0x7f8f840a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 --2- 172.21.15.79:0/3868623024 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 0x7f8f840a3750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009870 tx=0x7f8f7401c500 comp rx=0 tx=0).stop 2024-04-23T16:39:47.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 -- 172.21.15.79:0/3868623024 shutdown_connections 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 --2- 172.21.15.79:0/3868623024 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f840a3370 0x7f8f840a3750 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 --2- 172.21.15.79:0/3868623024 >> v2:172.21.15.79:3300/0 conn(0x7f8f8406ca30 0x7f8f840a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 -- 172.21.15.79:0/3868623024 >> 172.21.15.79:0/3868623024 conn(0x7f8f840aa3e0 msgr2=0x7f8f840a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 -- 172.21.15.79:0/3868623024 shutdown_connections 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.749+0000 7f8f8b369640 1 -- 172.21.15.79:0/3868623024 wait complete. 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient: build_initial_monmap 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient: monmap: 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:47.762 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 Processor -- start 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 -- start start 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient: init 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 0x7f8f8413e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): picked mon.noname-c con 0x7f8f8406ca30 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f840a3370 0x7f8f8413ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): picked mon.noname-b con 0x7f8f840a3370 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): picked mon.noname-a con 0x7f8f840a47e0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84142490 con 0x7f8f840a47e0 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84142310 con 0x7f8f840a3370 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f8f84142610 con 0x7f8f8406ca30 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f8b369640 10 monclient(hunting): authenticate will time out at 941.945269s 2024-04-23T16:39:47.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 0x7f8f8413e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35690/0 (socket says 172.21.15.79:35690) 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 -- 172.21.15.79:0/4224125737 learned_addr learned my addr 172.21.15.79:0/4224125737 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 10 monclient(hunting): get_auth_request con 0x7f8f840a47e0 auth_method 0 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 0x7f8f8413e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59906/0 (socket says 172.21.15.79:59906) 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): get_auth_request con 0x7f8f8406ca30 auth_method 0 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 -- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 msgr2=0x7f8f841253b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-23T16:39:47.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 -- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 msgr2=0x7f8f841253b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89906640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f88904640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f840a3370 0x7f8f8413ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 -- 172.21.15.79:0/4224125737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f840a3370 msgr2=0x7f8f8413ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f840a3370 0x7f8f8413ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 -- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 msgr2=0x7f8f841253b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient: _finish_hunting 0 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 monclient: found mon.noname-c 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:47.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:47.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 -- 172.21.15.79:0/4224125737 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f841292f0 con 0x7f8f8406ca30 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 -- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 msgr2=0x7f8f8413e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.753+0000 7f8f89105640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 0x7f8f8413e4c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000e8e0 tx=0x7f8f8000eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80017070 con 0x7f8f8406ca30 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: dump: 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.769 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f8000caf0 con 0x7f8f8406ca30 2024-04-23T16:39:47.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80013470 con 0x7f8f8406ca30 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 10 monclient: dump: 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:47.771 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 5 monclient: authenticate success, global_id 4265 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 monclient: _renew_subs 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f840a4000 con 0x7f8f8406ca30 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f7b7fe640 4 set_mon_vals no callback set 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 monclient: _renew_subs 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f841278e0 con 0x7f8f8406ca30 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f84068ae0 con 0x7f8f8406ca30 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f8f8001a070 con 0x7f8f8406ca30 2024-04-23T16:39:47.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 1 --2- 172.21.15.79:0/4224125737 >> v2:172.21.15.204:6812/722020384 conn(0x7f8f68036430 0x7f8f680388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f8f80049300 con 0x7f8f8406ca30 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f88904640 1 --2- 172.21.15.79:0/4224125737 >> v2:172.21.15.204:6812/722020384 conn(0x7f8f68036430 0x7f8f680388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.757+0000 7f8f88904640 10 monclient: get_auth_request con 0x7f8f68036430 auth_method 0 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.761+0000 7f8f88904640 1 --2- 172.21.15.79:0/4224125737 >> v2:172.21.15.204:6812/722020384 conn(0x7f8f68036430 0x7f8f680388f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f8406d150 tx=0x7f8f74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.761+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8f80013e10 con 0x7f8f8406ca30 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.761+0000 7f8f79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:47.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.761+0000 7f8f79ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f8b369640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f8b369640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f8b369640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8f84072040 con 0x7f8f8406ca30 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f79ffb640 1 -- 172.21.15.79:0/4224125737 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8f800481f0 con 0x7f8f8406ca30 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f79ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-23T16:39:47.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.941+0000 7f8f79ffb640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:47.947 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:47.947 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 >> v2:172.21.15.204:6812/722020384 conn(0x7f8f68036430 msgr2=0x7f8f680388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 --2- 172.21.15.79:0/4224125737 >> v2:172.21.15.204:6812/722020384 conn(0x7f8f68036430 0x7f8f680388f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f8406d150 tx=0x7f8f74008040 comp rx=0 tx=0).stop 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 10 monclient: shutdown 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 msgr2=0x7f8f8413e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 0x7f8f8413e4c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000e8e0 tx=0x7f8f8000eda0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 shutdown_connections 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 --2- 172.21.15.79:0/4224125737 >> v2:172.21.15.204:6812/722020384 conn(0x7f8f68036430 0x7f8f680388f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f840a47e0 0x7f8f841253b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f840a3370 0x7f8f8413ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 --2- 172.21.15.79:0/4224125737 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8f8406ca30 0x7f8f8413e4c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 >> 172.21.15.79:0/4224125737 conn(0x7f8f840aa3e0 msgr2=0x7f8f8409e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 shutdown_connections 2024-04-23T16:39:47.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:47.945+0000 7f8f8b369640 1 -- 172.21.15.79:0/4224125737 wait complete. 2024-04-23T16:39:47.963 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-23T16:39:47.963 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-23T16:39:47.977 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-23T16:39:47.978 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T16:39:48.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.077+0000 7efd9a46d640 10 monclient: get_monmap_and_config 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.077+0000 7efd9a46d640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.077+0000 7efd9a46d640 10 monclient: monmap: 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:48.084755+0000 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:48.084755+0000 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:48.083 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:48.084 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:48.084 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:48.084 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 Processor -- start 2024-04-23T16:39:48.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 -- start start 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient: init 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7efd9406ca30 0x7efd940a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): picked mon.noname-a con 0x7efd9406ca30 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd940a32a0 0x7efd940a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): picked mon.noname-b con 0x7efd940a32a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7efd941375a0 0x7efd94137a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): picked mon.noname-c con 0x7efd941375a0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7efd9406ce10 con 0x7efd9406ca30 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7efd940a17b0 con 0x7efd941375a0 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efd940a14d0 con 0x7efd940a32a0 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd9a46d640 10 monclient(hunting): authenticate will time out at 942.272866s 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7efd9406ca30 0x7efd940a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd940a32a0 0x7efd940a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7efd9406ca30 0x7efd940a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35702/0 (socket says 172.21.15.79:35702) 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 -- 172.21.15.79:0/1213561629 learned_addr learned my addr 172.21.15.79:0/1213561629 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): get_auth_request con 0x7efd9406ca30 auth_method 0 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): get_auth_request con 0x7efd940a32a0 auth_method 0 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd98a0a640 1 --2- 172.21.15.79:0/1213561629 >> v2:172.21.15.204:3300/0 conn(0x7efd941375a0 0x7efd94137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd98a0a640 10 monclient(hunting): get_auth_request con 0x7efd941375a0 auth_method 0 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd98a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd98a0a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd98a0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 -- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd940a32a0 msgr2=0x7efd940a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 --2- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd940a32a0 0x7efd940a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 -- 172.21.15.79:0/1213561629 >> v2:172.21.15.204:3300/0 conn(0x7efd941375a0 msgr2=0x7efd94137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 --2- 172.21.15.79:0/1213561629 >> v2:172.21.15.204:3300/0 conn(0x7efd941375a0 0x7efd94137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 monclient: found mon.noname-a 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.081+0000 7efd93fff640 1 -- 172.21.15.79:0/1213561629 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd940a1a90 con 0x7efd9406ca30 2024-04-23T16:39:48.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd93fff640 1 -- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd9406ca30 msgr2=0x7efd940a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd93fff640 1 --2- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd9406ca30 0x7efd940a4c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efd940a7c20 tx=0x7efd8401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7dad6dba1377960 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 1 -- 172.21.15.79:0/1213561629 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd840b0070 con 0x7efd9406ca30 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: dump: 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.098 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 1 -- 172.21.15.79:0/1213561629 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd840ab600 con 0x7efd9406ca30 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 5 monclient: authenticate success, global_id 4326 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 1 -- 172.21.15.79:0/1213561629 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd840ab9d0 con 0x7efd9406ca30 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 4 set_mon_vals no callback set 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd937fe640 10 monclient: dump: 2024-04-23T16:39:48.099 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: shutdown 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd9406ca30 msgr2=0x7efd940a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd9406ca30 0x7efd940a4c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efd940a7c20 tx=0x7efd8401c3e0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- 172.21.15.79:0/1213561629 shutdown_connections 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd940a32a0 0x7efd940a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1213561629 >> v2:172.21.15.204:3300/0 conn(0x7efd941375a0 0x7efd94137a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1213561629 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd9406ca30 0x7efd940a4c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- 172.21.15.79:0/1213561629 >> 172.21.15.79:0/1213561629 conn(0x7efd940aa370 msgr2=0x7efd940a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- 172.21.15.79:0/1213561629 shutdown_connections 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- 172.21.15.79:0/1213561629 wait complete. 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: monmap: 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.101 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 Processor -- start 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- start start 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: init 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd941cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): picked mon.noname-b con 0x7efd9406ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): picked mon.noname-a con 0x7efd940a32a0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd9413a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): picked mon.noname-c con 0x7efd941375a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd94141270 con 0x7efd940a32a0 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd941410f0 con 0x7efd9406ca30 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efd941413f0 con 0x7efd941375a0 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.085+0000 7efd9a46d640 10 monclient(hunting): authenticate will time out at 942.279968s 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd9413a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35718/0 (socket says 172.21.15.79:35718) 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 learned_addr learned my addr 172.21.15.79:0/1897573424 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): get_auth_request con 0x7efd940a32a0 auth_method 0 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): get_auth_request con 0x7efd941375a0 auth_method 0 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd93fff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd941cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd93fff640 10 monclient(hunting): get_auth_request con 0x7efd9406ca30 auth_method 0 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd93fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd93fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 msgr2=0x7efd9413a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd9413a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 msgr2=0x7efd941cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd941cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 monclient: found mon.noname-a 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9413e370 con 0x7efd940a32a0 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd93fff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd941cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd98a0a640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd9413a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 msgr2=0x7efd94134500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e9b0 tx=0x7efd7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c017070 con 0x7efd940a32a0 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: dump: 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c00cbb0 con 0x7efd940a32a0 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 5 monclient: authenticate success, global_id 4329 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 monclient: _renew_subs 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd940a4060 con 0x7efd940a32a0 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c013470 con 0x7efd940a32a0 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: dump: 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.111 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 monclient: _renew_subs 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd92ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd9413cf80 con 0x7efd940a32a0 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e9b0 tx=0x7efd7c00ee70 comp rx=0 tx=0).queue_received will delay after 1.71389e+09 on 0x7efd7c049050 osd_map(15..15 src has 1..15) 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd940692e0 con 0x7efd940a32a0 2024-04-23T16:39:48.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7efd7c01a070 con 0x7efd940a32a0 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 msgr2=0x7efd94134500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 msgr2=0x7efd94134500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e9b0 tx=0x7efd7c00ee70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e9b0 tx=0x7efd7c00ee70 comp rx=0 tx=0).stop 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 --2- 172.21.15.79:0/1897573424 >> v2:172.21.15.204:6812/722020384 conn(0x7efd78036410 0x7efd780388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 msgr2=0x7efd94134500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd940a32a0 0x7efd94134500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd7803afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): picked mon.c con 0x7efd941375a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 0x7efd7803dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): picked mon.a con 0x7efd7803b840 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:48.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd78040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): picked mon.b con 0x7efd9406ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd78041130 con 0x7efd7803b840 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd780412b0 con 0x7efd9406ca30 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7efd78041430 con 0x7efd941375a0 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.089+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (unknown 0 0 0) 0x7efd7c049050 con 0x7efd940a32a0 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 0x7efd7803dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): get_auth_request con 0x7efd7803b840 auth_method 0 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd7803afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): get_auth_request con 0x7efd941375a0 auth_method 0 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd93fff640 1 --2- 172.21.15.79:0/1897573424 >> v2:172.21.15.204:6812/722020384 conn(0x7efd78036410 0x7efd780388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd93fff640 10 monclient(hunting): get_auth_request con 0x7efd78036410 auth_method 0 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd93fff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd78040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient(hunting): handle_auth_done global_id 4329 payload 293 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 msgr2=0x7efd7803afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd7803afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 msgr2=0x7efd78040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd78040720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 monclient: found mon.a 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7efd780415e0 con 0x7efd7803b840 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8c0076d0 con 0x7efd7803b840 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd93fff640 10 monclient: get_auth_request con 0x7efd9406ca30 auth_method 0 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd8bfff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd7803afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd93fff640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd78040720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 msgr2=0x7efd7803dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd98a0a640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 0x7efd7803dcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd8c007d40 tx=0x7efd7c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.093+0000 7efd93fff640 1 --2- 172.21.15.79:0/1897573424 >> v2:172.21.15.204:6812/722020384 conn(0x7efd78036410 0x7efd780388d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd9406d040 tx=0x7efd84008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c007350 con 0x7efd7803b840 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 10 monclient: dump: 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.119 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd8c01c070 con 0x7efd7803b840 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd927fc640 4 set_mon_vals no callback set 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7efd8c010cb0 con 0x7efd7803b840 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.097+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 4 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7efd8c0486a0 con 0x7efd7803b840 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.101+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efd8c0465e0 con 0x7efd7803b840 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.101+0000 7efd917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.101+0000 7efd917fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:48.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:48.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:48.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd9413c960 con 0x7efd7803b840 2024-04-23T16:39:48.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd917fa640 1 -- 172.21.15.79:0/1897573424 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10799 (secure 0 0 0) 0x7efd8c046960 con 0x7efd7803b840 2024-04-23T16:39:48.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:39:48.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd917fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:48.253 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:48.254 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":15,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:39:44.700457+0000","last_up_change":"2024-04-23T16:39:41.690496+0000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:39:41.354561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-23T16:39:42.435523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":987647954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":987647954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":987647954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":987647954}]},"public_addr":"172.21.15.79:6802/987647954","cluster_addr":"172.21.15.79:6805/987647954","heartbeat_back_addr":"172.21.15.79:6810/987647954","heartbeat_front_addr":"172.21.15.79:6808/987647954","state":["exists","up"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3615156451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":3615156451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":3615156451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":3615156451}]},"public_addr":"172.21.15.79:6803/3615156451","cluster_addr":"172.21.15.79:6806/3615156451","heartbeat_back_addr":"172.21.15.79:6811/3615156451","heartbeat_front_addr":"172.21.15.79:6809/3615156451","state":["exists","up"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3678369310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":3678369310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":3678369310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":3678369310}]},"public_addr":"172.21.15.79:6800/3678369310","cluster_addr":"172.21.15.79:6801/3678369310","heartbeat_back_addr":"172.21.15.79:6807/3678369310","heartbeat_front_addr":"172.21.15.79:6804/3678369310","state":["exists","up"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3194378740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":3194378740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":3194378740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3194378740}]},"public_addr":"172.21.15.204:6800/3194378740","cluster_addr":"172.21.15.204:6801/3194378740","heartbeat_back_addr":"172.21.15.204:6803/3194378740","heartbeat_front_addr":"172.21.15.204:6802/3194378740","state":["exists","up"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3419064638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":3419064638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":3419064638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3419064638}]},"public_addr":"172.21.15.204:6804/3419064638","cluster_addr":"172.21.15.204:6805/3419064638","heartbeat_back_addr":"172.21.15.204:6807/3419064638","heartbeat_front_addr":"172.21.15.204:6806/3419064638","state":["exists","up"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":3961528928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":3961528928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":3961528928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3961528928}]},"public_addr":"172.21.15.204:6808/3961528928","cluster_addr":"172.21.15.204:6809/3961528928","heartbeat_back_addr":"172.21.15.204:6811/3961528928","heartbeat_front_addr":"172.21.15.204:6810/3961528928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.690894+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.747794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.664067+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.480934+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.716414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.584524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:39:48.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 >> v2:172.21.15.204:6812/722020384 conn(0x7efd78036410 msgr2=0x7efd780388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1897573424 >> v2:172.21.15.204:6812/722020384 conn(0x7efd78036410 0x7efd780388d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd9406d040 tx=0x7efd84008040 comp rx=0 tx=0).stop 2024-04-23T16:39:48.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 10 monclient: shutdown 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 msgr2=0x7efd7803dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 0x7efd7803dcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd8c007d40 tx=0x7efd7c009190 comp rx=0 tx=0).stop 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 shutdown_connections 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd9406ca30 0x7efd78040720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd7803b840 0x7efd7803dcd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1897573424 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7efd941375a0 0x7efd7803afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 --2- 172.21.15.79:0/1897573424 >> v2:172.21.15.204:6812/722020384 conn(0x7efd78036410 0x7efd780388d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 >> 172.21.15.79:0/1897573424 conn(0x7efd940aa370 msgr2=0x7efd940a8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 shutdown_connections 2024-04-23T16:39:48.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.249+0000 7efd9a46d640 1 -- 172.21.15.79:0/1897573424 wait complete. 2024-04-23T16:39:48.271 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:39:41.354561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-23T16:39:42.435523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T16:39:48.271 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T16:39:48.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient: get_monmap_and_config 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient: monmap: 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:48.383714+0000 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:48.383714+0000 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:48.382 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:48.383 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:48.383 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:48.383 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 Processor -- start 2024-04-23T16:39:48.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 -- start start 2024-04-23T16:39:48.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient: init 2024-04-23T16:39:48.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-23T16:39:48.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe730134560 0x7fe730134940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): picked mon.noname-c con 0x7fe730134560 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7300a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): picked mon.noname-b con 0x7fe73009da60 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7300a1920 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe73009e8a0 con 0x7fe7300a1920 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe73009d8a0 con 0x7fe730134560 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fe73009e5c0 con 0x7fe73009da60 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe736689640 10 monclient(hunting): authenticate will time out at 942.571536s 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7300a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7300a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59924/0 (socket says 172.21.15.79:59924) 2024-04-23T16:39:48.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 1 -- 172.21.15.79:0/583725837 learned_addr learned my addr 172.21.15.79:0/583725837 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): get_auth_request con 0x7fe73009da60 auth_method 0 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72ffff640 1 --2- 172.21.15.79:0/583725837 >> v2:172.21.15.204:3300/0 conn(0x7fe730134560 0x7fe730134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72ffff640 10 monclient(hunting): get_auth_request con 0x7fe730134560 auth_method 0 2024-04-23T16:39:48.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.377+0000 7fe72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 1 -- 172.21.15.79:0/583725837 >> v2:172.21.15.204:3300/0 conn(0x7fe730134560 msgr2=0x7fe730134940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 1 --2- 172.21.15.79:0/583725837 >> v2:172.21.15.204:3300/0 conn(0x7fe730134560 0x7fe730134940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 1 -- 172.21.15.79:0/583725837 >> v1:172.21.15.204:6789/0 conn(0x7fe7300a1920 legacy=0x7fe73013b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 1 monclient: found mon.noname-b 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:48.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.381+0000 7fe72f7fe640 1 -- 172.21.15.79:0/583725837 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7300a2ea0 con 0x7fe73009da60 2024-04-23T16:39:48.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72f7fe640 1 -- 172.21.15.79:0/583725837 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 msgr2=0x7fe7300a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72f7fe640 1 --2- 172.21.15.79:0/583725837 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7300a13e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe724009870 tx=0x7fe72401c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2bb4637df146d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 1 -- 172.21.15.79:0/583725837 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7240b0070 con 0x7fe73009da60 2024-04-23T16:39:48.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: dump: 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 1 -- 172.21.15.79:0/583725837 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7240ab600 con 0x7fe73009da60 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 5 monclient: authenticate success, global_id 4274 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 4 set_mon_vals no callback set 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 1 -- 172.21.15.79:0/583725837 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7240ab9d0 con 0x7fe73009da60 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72effd640 10 monclient: dump: 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.396 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: shutdown 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- 172.21.15.79:0/583725837 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 msgr2=0x7fe7300a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 --2- 172.21.15.79:0/583725837 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7300a13e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe724009870 tx=0x7fe72401c200 comp rx=0 tx=0).stop 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- 172.21.15.79:0/583725837 shutdown_connections 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 --2- 172.21.15.79:0/583725837 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7300a13e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 --2- 172.21.15.79:0/583725837 >> v2:172.21.15.204:3300/0 conn(0x7fe730134560 0x7fe730134940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- 172.21.15.79:0/583725837 >> 172.21.15.79:0/583725837 conn(0x7fe7300a6250 msgr2=0x7fe7300a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- 172.21.15.79:0/583725837 shutdown_connections 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- 172.21.15.79:0/583725837 wait complete. 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: monmap: 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:48.398 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 Processor -- start 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- start start 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: init 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7301cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): picked mon.noname-c con 0x7fe73009da60 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7300a1920 0x7fe7301d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): picked mon.noname-b con 0x7fe7300a1920 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:48.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 0x7fe73013aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): picked mon.noname-a con 0x7fe730134560 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe73013d510 con 0x7fe730134560 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe73013d390 con 0x7fe7300a1920 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fe73013d690 con 0x7fe73009da60 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe736689640 10 monclient(hunting): authenticate will time out at 942.578931s 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 0x7fe73013aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7301cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 0x7fe73013aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35746/0 (socket says 172.21.15.79:35746) 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 -- 172.21.15.79:0/1302059105 learned_addr learned my addr 172.21.15.79:0/1302059105 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): get_auth_request con 0x7fe730134560 auth_method 0 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): get_auth_request con 0x7fe73009da60 auth_method 0 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72f7fe640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7300a1920 0x7fe7301d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 -- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 msgr2=0x7fe7301cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7301cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 -- 172.21.15.79:0/1302059105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7300a1920 msgr2=0x7fe7301d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7300a1920 0x7fe7301d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 monclient: found mon.noname-a 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe734c26640 1 -- 172.21.15.79:0/1302059105 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73011c410 con 0x7fe730134560 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72f7fe640 10 monclient: get_auth_request con 0x7fe7300a1920 auth_method 0 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.385+0000 7fe72f7fe640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7300a1920 0x7fe7301d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe734c26640 1 -- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 msgr2=0x7fe73013aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe734c26640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 0x7fe73013aa90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c7b0 tx=0x7fe72800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72800ac60 con 0x7fe730134560 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: dump: 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe72800ae00 con 0x7fe730134560 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728004bb0 con 0x7fe730134560 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 10 monclient: dump: 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.404 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72e7fc640 4 set_mon_vals no callback set 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 5 monclient: authenticate success, global_id 4332 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 monclient: _renew_subs 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73011ded0 con 0x7fe730134560 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 monclient: _renew_subs 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7301d0830 con 0x7fe730134560 2024-04-23T16:39:48.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7300692e0 con 0x7fe730134560 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fe728005030 con 0x7fe730134560 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 1 --2- 172.21.15.79:0/1302059105 >> v2:172.21.15.204:6812/722020384 conn(0x7fe71c036430 0x7fe71c0388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fe7280491c0 con 0x7fe730134560 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72ffff640 1 --2- 172.21.15.79:0/1302059105 >> v2:172.21.15.204:6812/722020384 conn(0x7fe71c036430 0x7fe71c0388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.389+0000 7fe72ffff640 10 monclient: get_auth_request con 0x7fe71c036430 auth_method 0 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.393+0000 7fe72ffff640 1 --2- 172.21.15.79:0/1302059105 >> v2:172.21.15.204:6812/722020384 conn(0x7fe71c036430 0x7fe71c0388f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe720007a60 tx=0x7fe720008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.393+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe728020070 con 0x7fe730134560 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.393+0000 7fe72cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.393+0000 7fe72cff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:48.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-23T16:39:48.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-23T16:39:48.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe73006ba20 con 0x7fe730134560 2024-04-23T16:39:48.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe72cff9640 1 -- 172.21.15.79:0/1302059105 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) ==== 93+0+10 (secure 0 0 0) 0x7fe72801a7a0 con 0x7fe730134560 2024-04-23T16:39:48.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe72cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-23T16:39:48.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe72cff9640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:48.549 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 >> v2:172.21.15.204:6812/722020384 conn(0x7fe71c036430 msgr2=0x7fe71c0388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 --2- 172.21.15.79:0/1302059105 >> v2:172.21.15.204:6812/722020384 conn(0x7fe71c036430 0x7fe71c0388f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe720007a60 tx=0x7fe720008040 comp rx=0 tx=0).stop 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 10 monclient: shutdown 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 msgr2=0x7fe73013aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 0x7fe73013aa90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c7b0 tx=0x7fe72800cc70 comp rx=0 tx=0).stop 2024-04-23T16:39:48.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 shutdown_connections 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 --2- 172.21.15.79:0/1302059105 >> v2:172.21.15.204:6812/722020384 conn(0x7fe71c036430 0x7fe71c0388f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe730134560 0x7fe73013aa90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7300a1920 0x7fe7301d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 --2- 172.21.15.79:0/1302059105 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fe73009da60 0x7fe7301cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 >> 172.21.15.79:0/1302059105 conn(0x7fe7300a6250 msgr2=0x7fe73009ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 shutdown_connections 2024-04-23T16:39:48.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.545+0000 7fe736689640 1 -- 172.21.15.79:0/1302059105 wait complete. 2024-04-23T16:39:48.566 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-23T16:39:48.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.669+0000 7fb2689fa640 10 monclient: get_monmap_and_config 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.669+0000 7fb2689fa640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.669+0000 7fb2689fa640 10 monclient: monmap: 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:48.676653+0000 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:48.676653+0000 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:48.675 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:48.676 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 1 Processor -- start 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 1 -- start start 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient: init 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb26406cab0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2640a3c30 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:48.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb264137b80 0x7fb2641400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb264137b80 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb264138590 con 0x7fb26406cab0 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2640a5750 con 0x7fb2640a3c30 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fb26406ce90 con 0x7fb264137b80 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb2689fa640 10 monclient(hunting): authenticate will time out at 942.864986s 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb264137b80 0x7fb2641400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb264137b80 0x7fb2641400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35748/0 (socket says 172.21.15.79:35748) 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 -- 172.21.15.79:0/2480204008 learned_addr learned my addr 172.21.15.79:0/2480204008 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): get_auth_request con 0x7fb264137b80 auth_method 0 2024-04-23T16:39:48.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb26259c640 1 -- 172.21.15.79:0/2480204008 >> v1:172.21.15.79:6789/0 conn(0x7fb26406cab0 legacy=0x7fb2640a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 -- 172.21.15.79:0/2480204008 >> v1:172.21.15.204:6789/0 conn(0x7fb2640a3c30 legacy=0x7fb26413c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 -- 172.21.15.79:0/2480204008 >> v1:172.21.15.79:6789/0 conn(0x7fb26406cab0 legacy=0x7fb2640a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 monclient: found mon.noname-a 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.673+0000 7fb262d9d640 1 -- 172.21.15.79:0/2480204008 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2640a17d0 con 0x7fb264137b80 2024-04-23T16:39:48.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb262d9d640 1 -- 172.21.15.79:0/2480204008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 msgr2=0x7fb2641400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb262d9d640 1 --2- 172.21.15.79:0/2480204008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb2641400f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009ed0 tx=0x7fb25c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5e3a4281103fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 1 -- 172.21.15.79:0/2480204008 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0a56e0 con 0x7fb264137b80 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: dump: 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 1 -- 172.21.15.79:0/2480204008 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb25c0a5880 con 0x7fb264137b80 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb2689fa640 5 monclient: authenticate success, global_id 4335 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb2689fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb2689fa640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 1 -- 172.21.15.79:0/2480204008 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0a5c50 con 0x7fb264137b80 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb2689fa640 4 set_mon_vals no callback set 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.677+0000 7fb26159a640 10 monclient: dump: 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.685 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.686 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient: shutdown 2024-04-23T16:39:48.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- 172.21.15.79:0/2480204008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 msgr2=0x7fb2641400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 --2- 172.21.15.79:0/2480204008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb2641400f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009ed0 tx=0x7fb25c01c930 comp rx=0 tx=0).stop 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- 172.21.15.79:0/2480204008 shutdown_connections 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 --2- 172.21.15.79:0/2480204008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb2641400f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- 172.21.15.79:0/2480204008 >> 172.21.15.79:0/2480204008 conn(0x7fb2640aa3d0 msgr2=0x7fb2640a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- 172.21.15.79:0/2480204008 shutdown_connections 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- 172.21.15.79:0/2480204008 wait complete. 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient: monmap: 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.691 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 Processor -- start 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- start start 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient: init 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb26406cab0 0x7fb264130fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb26406cab0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2640a3c30 0x7fb2641314f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb2640a3c30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:48.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb264131a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb264137b80 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb264143240 con 0x7fb264137b80 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641430c0 con 0x7fb2640a3c30 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fb2641433c0 con 0x7fb26406cab0 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb2689fa640 10 monclient(hunting): authenticate will time out at 942.874146s 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb264131a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb26406cab0 0x7fb264130fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb264131a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35758/0 (socket says 172.21.15.79:35758) 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 -- 172.21.15.79:0/4285901186 learned_addr learned my addr 172.21.15.79:0/4285901186 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): get_auth_request con 0x7fb264137b80 auth_method 0 2024-04-23T16:39:48.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): get_auth_request con 0x7fb26406cab0 auth_method 0 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb26259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-04-23T16:39:48.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 -- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb26406cab0 msgr2=0x7fb264130fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb26406cab0 0x7fb264130fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 -- 172.21.15.79:0/4285901186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2640a3c30 msgr2=0x7fb2641314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2640a3c30 0x7fb2641314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 monclient: found mon.noname-a 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.681+0000 7fb262d9d640 1 -- 172.21.15.79:0/4285901186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb264133e20 con 0x7fb264137b80 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb262d9d640 1 -- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 msgr2=0x7fb264131a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb262d9d640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb264131a30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb25c019540 tx=0x7fb25c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0bc020 con 0x7fb264137b80 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: dump: 2024-04-23T16:39:48.695 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb25c0b8880 con 0x7fb264137b80 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 5 monclient: authenticate success, global_id 4341 2024-04-23T16:39:48.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 monclient: _renew_subs 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb264132970 con 0x7fb264137b80 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0b8c70 con 0x7fb264137b80 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 10 monclient: dump: 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.697 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb260d99640 4 set_mon_vals no callback set 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 monclient: _renew_subs 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2640a99e0 con 0x7fb264137b80 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2640692e0 con 0x7fb264137b80 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fb25c0bf020 con 0x7fb264137b80 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 1 --2- 172.21.15.79:0/4285901186 >> v2:172.21.15.204:6812/722020384 conn(0x7fb244036490 0x7fb244038950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fb25c0e3db0 con 0x7fb264137b80 2024-04-23T16:39:48.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb26259c640 1 --2- 172.21.15.79:0/4285901186 >> v2:172.21.15.204:6812/722020384 conn(0x7fb244036490 0x7fb244038950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.685+0000 7fb26259c640 10 monclient: get_auth_request con 0x7fb244036490 auth_method 0 2024-04-23T16:39:48.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.689+0000 7fb26259c640 1 --2- 172.21.15.79:0/4285901186 >> v2:172.21.15.204:6812/722020384 conn(0x7fb244036490 0x7fb244038950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb254007fd0 tx=0x7fb254007ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.689+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb25c0c6070 con 0x7fb264137b80 2024-04-23T16:39:48.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.689+0000 7fb252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.689+0000 7fb252ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:48.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-23T16:39:48.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-23T16:39:48.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fb264072040 con 0x7fb264137b80 2024-04-23T16:39:48.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb252ffd640 1 -- 172.21.15.79:0/4285901186 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) ==== 92+0+10 (secure 0 0 0) 0x7fb25c016640 con 0x7fb264137b80 2024-04-23T16:39:48.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-23T16:39:48.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb252ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:48.845 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 8 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 >> v2:172.21.15.204:6812/722020384 conn(0x7fb244036490 msgr2=0x7fb244038950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 --2- 172.21.15.79:0/4285901186 >> v2:172.21.15.204:6812/722020384 conn(0x7fb244036490 0x7fb244038950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb254007fd0 tx=0x7fb254007ce0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 10 monclient: shutdown 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 msgr2=0x7fb264131a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb264131a30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb25c019540 tx=0x7fb25c0196d0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 shutdown_connections 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 --2- 172.21.15.79:0/4285901186 >> v2:172.21.15.204:6812/722020384 conn(0x7fb244036490 0x7fb244038950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb264137b80 0x7fb264131a30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2640a3c30 0x7fb2641314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 --2- 172.21.15.79:0/4285901186 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb26406cab0 0x7fb264130fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 >> 172.21.15.79:0/4285901186 conn(0x7fb2640aa3d0 msgr2=0x7fb26412fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 shutdown_connections 2024-04-23T16:39:48.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.841+0000 7fb2689fa640 1 -- 172.21.15.79:0/4285901186 wait complete. 2024-04-23T16:39:48.861 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-23T16:39:48.861 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:39:48.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient: get_monmap_and_config 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient: monmap: 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:48.970613+0000 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:48.970613+0000 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:48.970 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 Processor -- start 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 -- start start 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient: init 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ff1ac06ca30 0x7ff1ac0a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ac06ca30 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:48.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1ac0a3370 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ac13c8c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ac0a2f20 con 0x7ff1ac0a3370 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ac0a1b50 con 0x7ff1ac06ca30 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ac06ce10 con 0x7ff1ac13c8c0 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b3a77640 10 monclient(hunting): authenticate will time out at 943.158465s 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac13ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59958/0 (socket says 172.21.15.79:59958) 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 -- 172.21.15.79:0/1398568812 learned_addr learned my addr 172.21.15.79:0/1398568812 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): get_auth_request con 0x7ff1ac13c8c0 auth_method 0 2024-04-23T16:39:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b1813640 1 --2- 172.21.15.79:0/1398568812 >> v2:172.21.15.204:3300/0 conn(0x7ff1ac06ca30 0x7ff1ac0a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b1813640 10 monclient(hunting): get_auth_request con 0x7ff1ac06ca30 auth_method 0 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b1813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b1813640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b1813640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 -- 172.21.15.79:0/1398568812 >> v2:172.21.15.204:3300/0 conn(0x7ff1ac06ca30 msgr2=0x7ff1ac0a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 --2- 172.21.15.79:0/1398568812 >> v2:172.21.15.204:3300/0 conn(0x7ff1ac06ca30 0x7ff1ac0a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 -- 172.21.15.79:0/1398568812 >> v1:172.21.15.204:6789/0 conn(0x7ff1ac0a3370 legacy=0x7ff1ac0a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 monclient: found mon.noname-b 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:48.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.965+0000 7ff1b2014640 1 -- 172.21.15.79:0/1398568812 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac0a5860 con 0x7ff1ac13c8c0 2024-04-23T16:39:48.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b2014640 1 -- 172.21.15.79:0/1398568812 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 msgr2=0x7ff1ac13ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b2014640 1 --2- 172.21.15.79:0/1398568812 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac13ecb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009e50 tx=0x7ff1a401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c47a9b8d8c5ab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 1 -- 172.21.15.79:0/1398568812 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40a55c0 con 0x7ff1ac13c8c0 2024-04-23T16:39:48.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:48.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: dump: 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 1 -- 172.21.15.79:0/1398568812 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a40a5760 con 0x7ff1ac13c8c0 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 5 monclient: authenticate success, global_id 4283 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 1 -- 172.21.15.79:0/1398568812 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40a5b30 con 0x7ff1ac13c8c0 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 4 set_mon_vals no callback set 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b0811640 10 monclient: dump: 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.981 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 10 monclient: shutdown 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 -- 172.21.15.79:0/1398568812 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 msgr2=0x7ff1ac13ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/1398568812 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac13ecb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009e50 tx=0x7ff1a401c940 comp rx=0 tx=0).stop 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 -- 172.21.15.79:0/1398568812 shutdown_connections 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/1398568812 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac13ecb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/1398568812 >> v2:172.21.15.204:3300/0 conn(0x7ff1ac06ca30 0x7ff1ac0a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 -- 172.21.15.79:0/1398568812 >> 172.21.15.79:0/1398568812 conn(0x7ff1ac0aa3e0 msgr2=0x7ff1ac0a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 -- 172.21.15.79:0/1398568812 shutdown_connections 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.969+0000 7ff1b3a77640 1 -- 172.21.15.79:0/1398568812 wait complete. 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient: build_initial_monmap 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient: monmap: 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:48.983 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 Processor -- start 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 -- start start 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient: init 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1ac06ca30 0x7ff1ac124d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ac06ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 0x7ff1ac13b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ac0a3370 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac1cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ac13c8c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac141da0 con 0x7ff1ac0a3370 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac141c20 con 0x7ff1ac06ca30 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ac141f20 con 0x7ff1ac13c8c0 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b3a77640 10 monclient(hunting): authenticate will time out at 943.165303s 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac1cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 0x7ff1ac13b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac1cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59972/0 (socket says 172.21.15.79:59972) 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 1 -- 172.21.15.79:0/2774493487 learned_addr learned my addr 172.21.15.79:0/2774493487 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): get_auth_request con 0x7ff1ac13c8c0 auth_method 0 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): get_auth_request con 0x7ff1ac0a3370 auth_method 0 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1813640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1ac06ca30 0x7ff1ac124d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 -- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 msgr2=0x7ff1ac1cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac1cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 -- 172.21.15.79:0/2774493487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1ac06ca30 msgr2=0x7ff1ac124d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1ac06ca30 0x7ff1ac124d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient: _finish_hunting 0 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 monclient: found mon.noname-a 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b1012640 1 -- 172.21.15.79:0/2774493487 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac122db0 con 0x7ff1ac0a3370 2024-04-23T16:39:48.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.973+0000 7ff1b2014640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac1cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:48.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b1012640 1 -- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 msgr2=0x7ff1ac13b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:48.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b1012640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 0x7ff1ac13b9b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c3e0 tx=0x7ff19c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0119a0 con 0x7ff1ac0a3370 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: dump: 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.990 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: _finish_auth 0 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: _check_auth_tickets 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 5 monclient: authenticate success, global_id 4344 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19c017070 con 0x7ff1ac0a3370 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 monclient: _renew_subs 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac123fe0 con 0x7ff1ac0a3370 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c011d80 con 0x7ff1ac0a3370 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:48.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 10 monclient: dump: 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 monclient: _renew_subs 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac125d80 con 0x7ff1ac0a3370 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a3fff640 4 set_mon_vals no callback set 2024-04-23T16:39:48.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ac13e6b0 con 0x7ff1ac0a3370 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7ff19c01c460 con 0x7ff1ac0a3370 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 1 --2- 172.21.15.79:0/2774493487 >> v2:172.21.15.204:6812/722020384 conn(0x7ff1900363e0 0x7ff1900388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7ff19c048d50 con 0x7ff1ac0a3370 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b1813640 1 --2- 172.21.15.79:0/2774493487 >> v2:172.21.15.204:6812/722020384 conn(0x7ff1900363e0 0x7ff1900388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.977+0000 7ff1b1813640 10 monclient: get_auth_request con 0x7ff1900363e0 auth_method 0 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.981+0000 7ff1b1813640 1 --2- 172.21.15.79:0/2774493487 >> v2:172.21.15.204:6812/722020384 conn(0x7ff1900363e0 0x7ff1900388a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009670 tx=0x7ff1a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.981+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff19c01a520 con 0x7ff1ac0a3370 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.981+0000 7ff1a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:48.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:48.981+0000 7ff1a27fc640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:49.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.125+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1ac068ae0 con 0x7ff1900363e0 2024-04-23T16:39:49.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1a27fc640 1 -- 172.21.15.79:0/2774493487 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33856 (secure 0 0 0) 0x7ff1ac068ae0 con 0x7ff1900363e0 2024-04-23T16:39:49.135 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:49.135 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:39:49.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 >> v2:172.21.15.204:6812/722020384 conn(0x7ff1900363e0 msgr2=0x7ff1900388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/2774493487 >> v2:172.21.15.204:6812/722020384 conn(0x7ff1900363e0 0x7ff1900388a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009670 tx=0x7ff1a8008040 comp rx=0 tx=0).stop 2024-04-23T16:39:49.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 10 monclient: shutdown 2024-04-23T16:39:49.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 msgr2=0x7ff1ac13b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 0x7ff1ac13b9b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c3e0 tx=0x7ff19c00c8a0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 shutdown_connections 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/2774493487 >> v2:172.21.15.204:6812/722020384 conn(0x7ff1900363e0 0x7ff1900388a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7ff1ac13c8c0 0x7ff1ac1cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1ac0a3370 0x7ff1ac13b9b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 --2- 172.21.15.79:0/2774493487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1ac06ca30 0x7ff1ac124d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 >> 172.21.15.79:0/2774493487 conn(0x7ff1ac0aa3e0 msgr2=0x7ff1ac13e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 shutdown_connections 2024-04-23T16:39:49.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.133+0000 7ff1b3a77640 1 -- 172.21.15.79:0/2774493487 wait complete. 2024-04-23T16:39:49.151 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T16:39:47.298131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163004,"kb_used_data":848,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068036,"statfs":{"total":579820584960,"available":579653668864,"internally_reserved":0,"allocated":868352,"data_stored":1274966,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888464+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:39:44.888464+0000","last_peered":"2024-04-23T16:39:44.888464+0000","last_clean":"2024-04-23T16:39:44.888464+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:39:44.888464+0000","last_undegraded":"2024-04-23T16:39:44.888464+0000","last_fullsized":"2024-04-23T16:39:44.888464+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.705938+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:39:44.705938+0000","last_peered":"2024-04-23T16:39:44.705938+0000","last_clean":"2024-04-23T16:39:44.705938+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:39:44.705938+0000","last_undegraded":"2024-04-23T16:39:44.705938+0000","last_fullsized":"2024-04-23T16:39:44.705938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888524+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:39:44.888524+0000","last_peered":"2024-04-23T16:39:44.888524+0000","last_clean":"2024-04-23T16:39:44.888524+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:39:44.888524+0000","last_undegraded":"2024-04-23T16:39:44.888524+0000","last_fullsized":"2024-04-23T16:39:44.888524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.707572+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:39:44.707572+0000","last_peered":"2024-04-23T16:39:44.707572+0000","last_clean":"2024-04-23T16:39:44.707572+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:39:44.707572+0000","last_undegraded":"2024-04-23T16:39:44.707572+0000","last_fullsized":"2024-04-23T16:39:44.707572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888798+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:39:44.888798+0000","last_peered":"2024-04-23T16:39:44.888798+0000","last_clean":"2024-04-23T16:39:44.888798+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:39:44.888798+0000","last_undegraded":"2024-04-23T16:39:44.888798+0000","last_fullsized":"2024-04-23T16:39:44.888798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.887962+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:39:44.887962+0000","last_peered":"2024-04-23T16:39:44.887962+0000","last_clean":"2024-04-23T16:39:44.887962+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:39:44.887962+0000","last_undegraded":"2024-04-23T16:39:44.887962+0000","last_fullsized":"2024-04-23T16:39:44.887962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.706991+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:39:44.706991+0000","last_peered":"2024-04-23T16:39:44.706991+0000","last_clean":"2024-04-23T16:39:44.706991+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:39:44.706991+0000","last_undegraded":"2024-04-23T16:39:44.706991+0000","last_fullsized":"2024-04-23T16:39:44.706991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888472+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:39:44.888472+0000","last_peered":"2024-04-23T16:39:44.888472+0000","last_clean":"2024-04-23T16:39:44.888472+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:39:44.888472+0000","last_undegraded":"2024-04-23T16:39:44.888472+0000","last_fullsized":"2024-04-23T16:39:44.888472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888262+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:39:44.888262+0000","last_peered":"2024-04-23T16:39:44.888262+0000","last_clean":"2024-04-23T16:39:44.888262+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:39:44.888262+0000","last_undegraded":"2024-04-23T16:39:44.888262+0000","last_fullsized":"2024-04-23T16:39:44.888262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607554,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":176128,"data_stored":563091,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607554,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":34163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":567135,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:39:49.152 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient: get_monmap_and_config 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient: build_initial_monmap 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient: monmap: 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:49.260030+0000 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:49.260030+0000 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:49.259 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 Processor -- start 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 -- start start 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient: init 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff8135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): picked mon.noname-b con 0x7f9ff8135a20 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9ff809f430 0x7f9ff809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): picked mon.noname-c con 0x7f9ff809f430 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f9ff8135300 0x7f9ff809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): picked mon.noname-a con 0x7f9ff8135300 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff8072f80 con 0x7f9ff8135300 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff809fb10 con 0x7f9ff809f430 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff809fdf0 con 0x7f9ff8135a20 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.253+0000 7f9ffd269640 10 monclient(hunting): authenticate will time out at 943.447814s 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f9ff8135300 0x7f9ff809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff8135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f9ff8135300 0x7f9ff809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35778/0 (socket says 172.21.15.79:35778) 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2196025097 learned_addr learned my addr 172.21.15.79:0/2196025097 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:49.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): get_auth_request con 0x7f9ff8135300 auth_method 0 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): get_auth_request con 0x7f9ff8135a20 auth_method 0 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff67fc640 1 --2- 172.21.15.79:0/2196025097 >> v2:172.21.15.204:3300/0 conn(0x7f9ff809f430 0x7f9ff809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff67fc640 10 monclient(hunting): get_auth_request con 0x7f9ff809f430 auth_method 0 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff67fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 msgr2=0x7f9ff8135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff8135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2196025097 >> v2:172.21.15.204:3300/0 conn(0x7f9ff809f430 msgr2=0x7f9ff809ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2196025097 >> v2:172.21.15.204:3300/0 conn(0x7f9ff809f430 0x7f9ff809ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 monclient: found mon.noname-a 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2196025097 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff806de30 con 0x7f9ff8135300 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff67fc640 1 --2- 172.21.15.79:0/2196025097 >> v2:172.21.15.204:3300/0 conn(0x7f9ff809f430 0x7f9ff809ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T16:39:49.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.257+0000 7f9ff6ffd640 1 --2- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff8135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 msgr2=0x7f9ff809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff809bf40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806cba0 tx=0x7f9ff001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421daec04d020dea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 1 -- 172.21.15.79:0/2196025097 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00a5530 con 0x7f9ff8135300 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: dump: 2024-04-23T16:39:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: _finish_auth 0 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 1 -- 172.21.15.79:0/2196025097 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff00a56d0 con 0x7f9ff8135300 2024-04-23T16:39:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 1 -- 172.21.15.79:0/2196025097 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00a5aa0 con 0x7f9ff8135300 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 5 monclient: authenticate success, global_id 4347 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 4 set_mon_vals no callback set 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ff5ffb640 10 monclient: dump: 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 10 monclient: shutdown 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 -- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 msgr2=0x7f9ff809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff809bf40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806cba0 tx=0x7f9ff001c8b0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 -- 172.21.15.79:0/2196025097 shutdown_connections 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff809bf40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2196025097 >> v2:172.21.15.204:3300/0 conn(0x7f9ff809f430 0x7f9ff809ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2196025097 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff8135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 -- 172.21.15.79:0/2196025097 >> 172.21.15.79:0/2196025097 conn(0x7f9ff80a25b0 msgr2=0x7f9ff80a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 -- 172.21.15.79:0/2196025097 shutdown_connections 2024-04-23T16:39:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 -- 172.21.15.79:0/2196025097 wait complete. 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 10 monclient: build_initial_monmap 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 10 monclient: monmap: 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 Processor -- start 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 1 -- start start 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.261+0000 7f9ffd269640 10 monclient: init 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ff809f430 0x7f9ff81cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): picked mon.noname-b con 0x7f9ff809f430 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff8134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): picked mon.noname-a con 0x7f9ff8135300 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff811c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): picked mon.noname-c con 0x7f9ff8135a20 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff813f0a0 con 0x7f9ff8135300 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff809c890 con 0x7f9ff809f430 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff813f220 con 0x7f9ff8135a20 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient(hunting): authenticate will time out at 943.456213s 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff8134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff8134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35784/0 (socket says 172.21.15.79:35784) 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 1 -- 172.21.15.79:0/2981973429 learned_addr learned my addr 172.21.15.79:0/2981973429 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): get_auth_request con 0x7f9ff8135300 auth_method 0 2024-04-23T16:39:49.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff811c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): get_auth_request con 0x7f9ff8135a20 auth_method 0 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-04-23T16:39:49.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2981973429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ff809f430 msgr2=0x7f9ff81cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff6ffd640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ff809f430 0x7f9ff81cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ff809f430 0x7f9ff81cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 msgr2=0x7f9ff8134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff8134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 monclient: found mon.noname-c 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2981973429 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff8072c70 con 0x7f9ff8135a20 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff67fc640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff8134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 -- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 msgr2=0x7f9ff811c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff77fe640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff811c100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806cf70 tx=0x7f9ff0019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00a5530 con 0x7f9ff8135a20 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: dump: 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.278 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff00b68a0 con 0x7f9ff8135a20 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00b6c90 con 0x7f9ff8135a20 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 5 monclient: authenticate success, global_id 4292 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _renew_subs 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff811d8f0 con 0x7f9ff8135a20 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ff57fa640 4 set_mon_vals no callback set 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9fdf7fe640 10 monclient: dump: 2024-04-23T16:39:49.279 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _renew_subs 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff811f340 con 0x7f9ff8135a20 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.269+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f9ff00be020 con 0x7f9ff8135a20 2024-04-23T16:39:49.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.265+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff80692e0 con 0x7f9ff8135a20 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.269+0000 7f9fdf7fe640 1 --2- 172.21.15.79:0/2981973429 >> v2:172.21.15.204:6812/722020384 conn(0x7f9fe4036430 0x7f9fe40388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.269+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f9ff00e32f0 con 0x7f9ff8135a20 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.269+0000 7f9ff6ffd640 1 --2- 172.21.15.79:0/2981973429 >> v2:172.21.15.204:6812/722020384 conn(0x7f9fe4036430 0x7f9fe40388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.269+0000 7f9ff6ffd640 10 monclient: get_auth_request con 0x7f9fe4036430 auth_method 0 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.269+0000 7f9ff6ffd640 1 --2- 172.21.15.79:0/2981973429 >> v2:172.21.15.204:6812/722020384 conn(0x7f9fe4036430 0x7f9fe40388f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009580 tx=0x7f9fec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.273+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9ff00c5020 con 0x7f9ff8135a20 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.273+0000 7f9fdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.273+0000 7f9fdf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:49.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.417+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ff81ce0a0 con 0x7f9fe4036430 2024-04-23T16:39:49.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9fdf7fe640 1 -- 172.21.15.79:0/2981973429 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33805 (secure 0 0 0) 0x7f9ff81ce0a0 con 0x7f9fe4036430 2024-04-23T16:39:49.425 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:39:49.426 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 >> v2:172.21.15.204:6812/722020384 conn(0x7f9fe4036430 msgr2=0x7f9fe40388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2981973429 >> v2:172.21.15.204:6812/722020384 conn(0x7f9fe4036430 0x7f9fe40388f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009580 tx=0x7f9fec008040 comp rx=0 tx=0).stop 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 10 monclient: shutdown 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 msgr2=0x7f9ff811c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff811c100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806cf70 tx=0x7f9ff0019650 comp rx=0 tx=0).stop 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 shutdown_connections 2024-04-23T16:39:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2981973429 >> v2:172.21.15.204:6812/722020384 conn(0x7f9fe4036430 0x7f9fe40388f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ff8135a20 0x7f9ff811c100 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff8135300 0x7f9ff8134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 --2- 172.21.15.79:0/2981973429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ff809f430 0x7f9ff81cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 >> 172.21.15.79:0/2981973429 conn(0x7f9ff80a25b0 msgr2=0x7f9ff8136430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 shutdown_connections 2024-04-23T16:39:49.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.421+0000 7f9ffd269640 1 -- 172.21.15.79:0/2981973429 wait complete. 2024-04-23T16:39:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-23T16:39:49.298670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163020,"kb_used_data":872,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566068020,"statfs":{"total":579820584960,"available":579653652480,"internally_reserved":0,"allocated":892928,"data_stored":1287098,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001138"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888464+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:39:44.888464+0000","last_peered":"2024-04-23T16:39:44.888464+0000","last_clean":"2024-04-23T16:39:44.888464+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:39:44.888464+0000","last_undegraded":"2024-04-23T16:39:44.888464+0000","last_fullsized":"2024-04-23T16:39:44.888464+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.705938+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:39:44.705938+0000","last_peered":"2024-04-23T16:39:44.705938+0000","last_clean":"2024-04-23T16:39:44.705938+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:39:44.705938+0000","last_undegraded":"2024-04-23T16:39:44.705938+0000","last_fullsized":"2024-04-23T16:39:44.705938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888524+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:39:44.888524+0000","last_peered":"2024-04-23T16:39:44.888524+0000","last_clean":"2024-04-23T16:39:44.888524+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:39:44.888524+0000","last_undegraded":"2024-04-23T16:39:44.888524+0000","last_fullsized":"2024-04-23T16:39:44.888524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.707572+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:39:44.707572+0000","last_peered":"2024-04-23T16:39:44.707572+0000","last_clean":"2024-04-23T16:39:44.707572+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:39:44.707572+0000","last_undegraded":"2024-04-23T16:39:44.707572+0000","last_fullsized":"2024-04-23T16:39:44.707572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888798+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:39:44.888798+0000","last_peered":"2024-04-23T16:39:44.888798+0000","last_clean":"2024-04-23T16:39:44.888798+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:39:44.888798+0000","last_undegraded":"2024-04-23T16:39:44.888798+0000","last_fullsized":"2024-04-23T16:39:44.888798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.887962+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:39:44.887962+0000","last_peered":"2024-04-23T16:39:44.887962+0000","last_clean":"2024-04-23T16:39:44.887962+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:39:44.887962+0000","last_undegraded":"2024-04-23T16:39:44.887962+0000","last_fullsized":"2024-04-23T16:39:44.887962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.706991+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:39:44.706991+0000","last_peered":"2024-04-23T16:39:44.706991+0000","last_clean":"2024-04-23T16:39:44.706991+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:39:44.706991+0000","last_undegraded":"2024-04-23T16:39:44.706991+0000","last_fullsized":"2024-04-23T16:39:44.706991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888472+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:39:44.888472+0000","last_peered":"2024-04-23T16:39:44.888472+0000","last_clean":"2024-04-23T16:39:44.888472+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:39:44.888472+0000","last_undegraded":"2024-04-23T16:39:44.888472+0000","last_fullsized":"2024-04-23T16:39:44.888472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-23T16:39:44.888262+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:39:44.888262+0000","last_peered":"2024-04-23T16:39:44.888262+0000","last_clean":"2024-04-23T16:39:44.888262+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:39:44.888262+0000","last_undegraded":"2024-04-23T16:39:44.888262+0000","last_fullsized":"2024-04-23T16:39:44.888262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607555,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":184320,"data_stored":567135,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607555,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":567135,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:39:49.442 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-23T16:39:49.443 DEBUG:tasks.mon_thrash:Yielding 2024-04-23T16:39:49.443 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T16:39:49.452 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T16:39:49.452 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:39:49.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20' 2024-04-23T16:39:49.453 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-23T16:39:49.453 INFO:tasks.mon_thrash.mon_thrasher:seed: 1713890389, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-04-23T16:39:49.453 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-23T16:39:49.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:39:49.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.557+0000 7f9eb30d2640 10 monclient: get_monmap_and_config 2024-04-23T16:39:49.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.557+0000 7f9eb30d2640 10 monclient: build_initial_monmap 2024-04-23T16:39:49.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.557+0000 7f9eb30d2640 10 monclient: monmap: 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:49.565290+0000 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:49.565290+0000 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:49.564 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 1 Processor -- start 2024-04-23T16:39:49.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 1 -- start start 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient: init 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9eac0a35c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9eac0a39a0 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9eac13b920 0x7f9eac13dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient(hunting): picked mon.noname-c con 0x7f9eac13b920 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac070700 con 0x7f9eac0a39a0 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac070900 con 0x7f9eac0a35c0 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9eac070490 con 0x7f9eac13b920 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb30d2640 10 monclient(hunting): authenticate will time out at 943.753474s 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eabfff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f9eac0a39a0 0x7f9eac0a3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:46164/0 (socket says 172.21.15.79:46164) 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eabfff640 1 -- 172.21.15.79:0/268181688 learned_addr learned my addr 172.21.15.79:0/268181688 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:49.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 1 --2- 172.21.15.79:0/268181688 >> v2:172.21.15.204:3300/0 conn(0x7f9eac13b920 0x7f9eac13dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): get_auth_request con 0x7f9eac13b920 auth_method 0 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 1 -- 172.21.15.79:0/268181688 >> v1:172.21.15.204:6789/0 conn(0x7f9eac0a35c0 legacy=0x7f9eac072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 1 -- 172.21.15.79:0/268181688 >> v1:172.21.15.79:6789/0 conn(0x7f9eac0a39a0 legacy=0x7f9eac0a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient: _finish_hunting 0 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 1 monclient: found mon.noname-c 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:49.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:39:49.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.561+0000 7f9eb166f640 1 -- 172.21.15.79:0/268181688 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac0a51b0 con 0x7f9eac13b920 2024-04-23T16:39:49.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb166f640 1 -- 172.21.15.79:0/268181688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac13b920 msgr2=0x7f9eac13dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb166f640 1 --2- 172.21.15.79:0/268181688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac13b920 0x7f9eac13dd10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009d90 tx=0x7f9ea401c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b547b59e18b762c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 1 -- 172.21.15.79:0/268181688 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40b2070 con 0x7f9eac13b920 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: dump: 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.570 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 1 -- 172.21.15.79:0/268181688 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea40a5a70 con 0x7f9eac13b920 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:49.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 5 monclient: authenticate success, global_id 4333 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 4 set_mon_vals no callback set 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 1 -- 172.21.15.79:0/268181688 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40a5e40 con 0x7f9eac13b920 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eab7fe640 10 monclient: dump: 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.572 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.573 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.573 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 10 monclient: shutdown 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 -- 172.21.15.79:0/268181688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac13b920 msgr2=0x7f9eac13dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/268181688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac13b920 0x7f9eac13dd10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009d90 tx=0x7f9ea401c7b0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 -- 172.21.15.79:0/268181688 shutdown_connections 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/268181688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac13b920 0x7f9eac13dd10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 -- 172.21.15.79:0/268181688 >> 172.21.15.79:0/268181688 conn(0x7f9eac0a83b0 msgr2=0x7f9eac0a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 -- 172.21.15.79:0/268181688 shutdown_connections 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 1 -- 172.21.15.79:0/268181688 wait complete. 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 10 monclient: build_initial_monmap 2024-04-23T16:39:49.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.565+0000 7f9eb30d2640 10 monclient: monmap: 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 Processor -- start 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 -- start start 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: init 2024-04-23T16:39:49.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac0a35c0 0x7f9eac135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): picked mon.noname-b con 0x7f9eac0a35c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 0x7f9eac09e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): picked mon.noname-c con 0x7f9eac0a39a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 0x7f9eac09e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): picked mon.noname-a con 0x7f9eac13b920 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac140e00 con 0x7f9eac13b920 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac140c80 con 0x7f9eac0a35c0 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac140f80 con 0x7f9eac0a39a0 2024-04-23T16:39:49.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient(hunting): authenticate will time out at 943.760618s 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 0x7f9eac09e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 0x7f9eac09e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 0x7f9eac09e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35790/0 (socket says 172.21.15.79:35790) 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 1 -- 172.21.15.79:0/4273538131 learned_addr learned my addr 172.21.15.79:0/4273538131 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 0x7f9eac09e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:59996/0 (socket says 172.21.15.79:59996) 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): get_auth_request con 0x7f9eac13b920 auth_method 0 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): get_auth_request con 0x7f9eac0a39a0 auth_method 0 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac0a35c0 0x7f9eac135180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): get_auth_request con 0x7f9eac0a35c0 auth_method 0 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb0e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 -- 172.21.15.79:0/4273538131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac0a35c0 msgr2=0x7f9eac135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac0a35c0 0x7f9eac135180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 -- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 msgr2=0x7f9eac09e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 0x7f9eac09e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 monclient: found mon.noname-c 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 -- 172.21.15.79:0/4273538131 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac0a09c0 con 0x7f9eac0a39a0 2024-04-23T16:39:49.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb166f640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 0x7f9eac09e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 -- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 msgr2=0x7f9eac09e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eabfff640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 0x7f9eac09e440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c440 tx=0x7f9ea000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0017070 con 0x7f9eac0a39a0 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: dump: 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.584 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 5 monclient: authenticate success, global_id 4304 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: _renew_subs 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac0a0580 con 0x7f9eac0a39a0 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea00070d0 con 0x7f9eac0a39a0 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00074c0 con 0x7f9eac0a39a0 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9ea97fa640 10 monclient: dump: 2024-04-23T16:39:49.585 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: _renew_subs 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eaaffd640 4 set_mon_vals no callback set 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.569+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac11e820 con 0x7f9eac0a39a0 2024-04-23T16:39:49.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9eb30d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9eb30d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9eb30d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac068ae0 con 0x7f9eac0a39a0 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f9ea001a070 con 0x7f9eac0a39a0 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9ea97fa640 1 --2- 172.21.15.79:0/4273538131 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e980363e0 0x7f9e980388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f9ea00483a0 con 0x7f9eac0a39a0 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9eb0e6e640 1 --2- 172.21.15.79:0/4273538131 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e980363e0 0x7f9e980388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.573+0000 7f9eb0e6e640 10 monclient: get_auth_request con 0x7f9e980363e0 auth_method 0 2024-04-23T16:39:49.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.577+0000 7f9eb0e6e640 1 --2- 172.21.15.79:0/4273538131 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e980363e0 0x7f9e980388a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007fd0 tx=0x7f9e9c007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.577+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9ea00076a0 con 0x7f9eac0a39a0 2024-04-23T16:39:49.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.577+0000 7f9ea97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.577+0000 7f9ea97fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.801+0000 7f9eb30d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.801+0000 7f9eb30d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.801+0000 7f9eb30d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.801+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9eac06dc30 con 0x7f9eac0a39a0 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9ea97fa640 1 -- 172.21.15.79:0/4273538131 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f9ea0012920 con 0x7f9eac0a39a0 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9ea97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9ea97fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:49.808 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e980363e0 msgr2=0x7f9e980388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/4273538131 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e980363e0 0x7f9e980388a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007fd0 tx=0x7f9e9c007e40 comp rx=0 tx=0).stop 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 10 monclient: shutdown 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 msgr2=0x7f9eac09e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 0x7f9eac09e440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c440 tx=0x7f9ea000c900 comp rx=0 tx=0).stop 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 shutdown_connections 2024-04-23T16:39:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/4273538131 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e980363e0 0x7f9e980388a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eac13b920 0x7f9eac09e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9eac0a39a0 0x7f9eac09e440 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 --2- 172.21.15.79:0/4273538131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac0a35c0 0x7f9eac135180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 >> 172.21.15.79:0/4273538131 conn(0x7f9eac0a83b0 msgr2=0x7f9eac0a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 shutdown_connections 2024-04-23T16:39:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.805+0000 7f9eb30d2640 1 -- 172.21.15.79:0/4273538131 wait complete. 2024-04-23T16:39:49.824 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-23T16:39:49.825 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-23T16:39:49.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-23T16:39:49.825 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-23T16:39:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient: get_monmap_and_config 2024-04-23T16:39:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient: build_initial_monmap 2024-04-23T16:39:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient: monmap: 2024-04-23T16:39:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:49.934430+0000 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:49.934430+0000 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:49.933 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 Processor -- start 2024-04-23T16:39:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 -- start start 2024-04-23T16:39:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient: init 2024-04-23T16:39:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-23T16:39:49.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcdd0134560 0x7fcdd0134940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): picked mon.noname-a con 0x7fcdd0134560 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdd009da60 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcdd00a1af0 0x7fcdd013c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): picked mon.noname-c con 0x7fcdd00a1af0 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdd009e8a0 con 0x7fcdd009da60 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd009d8a0 con 0x7fcdd0134560 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd009e5c0 con 0x7fcdd00a1af0 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdd5cd1640 10 monclient(hunting): authenticate will time out at 944.122350s 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcdd0134560 0x7fcdd0134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcdd0134560 0x7fcdd0134940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35806/0 (socket says 172.21.15.79:35806) 2024-04-23T16:39:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/876323170 learned_addr learned my addr 172.21.15.79:0/876323170 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): get_auth_request con 0x7fcdd0134560 auth_method 0 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcffff640 1 --2- 172.21.15.79:0/876323170 >> v2:172.21.15.204:3300/0 conn(0x7fcdd00a1af0 0x7fcdd013c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/876323170 >> v2:172.21.15.204:3300/0 conn(0x7fcdd00a1af0 msgr2=0x7fcdd013c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/876323170 >> v2:172.21.15.204:3300/0 conn(0x7fcdd00a1af0 0x7fcdd013c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/876323170 >> v1:172.21.15.204:6789/0 conn(0x7fcdd009da60 legacy=0x7fcdd00a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 monclient: found mon.noname-a 2024-04-23T16:39:49.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:49.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:49.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.929+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/876323170 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd00a2ea0 con 0x7fcdd0134560 2024-04-23T16:39:49.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/876323170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd0134560 msgr2=0x7fcdd0134940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/876323170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd0134560 0x7fcdd0134940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcdd006bf40 tx=0x7fcdc001c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4454e421b14862df server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 1 -- 172.21.15.79:0/876323170 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00b0070 con 0x7fcdd0134560 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: dump: 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.940 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: _finish_auth 0 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: _check_auth_tickets 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 5 monclient: authenticate success, global_id 4368 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 1 -- 172.21.15.79:0/876323170 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc00ab600 con 0x7fcdd0134560 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 4 set_mon_vals no callback set 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 1 -- 172.21.15.79:0/876323170 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00ab9d0 con 0x7fcdd0134560 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdce7fc640 10 monclient: dump: 2024-04-23T16:39:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 10 monclient: shutdown 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/876323170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd0134560 msgr2=0x7fcdd0134940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/876323170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd0134560 0x7fcdd0134940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcdd006bf40 tx=0x7fcdc001c1c0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/876323170 shutdown_connections 2024-04-23T16:39:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/876323170 >> v2:172.21.15.204:3300/0 conn(0x7fcdd00a1af0 0x7fcdd013c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/876323170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd0134560 0x7fcdd0134940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/876323170 >> 172.21.15.79:0/876323170 conn(0x7fcdd00a6250 msgr2=0x7fcdd00a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/876323170 shutdown_connections 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.933+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/876323170 wait complete. 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient: build_initial_monmap 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient: monmap: 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:49.943 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:49.944 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:49.944 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 Processor -- start 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 -- start start 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient: init 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdd009da60 0x7fcdd013b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): picked mon.noname-b con 0x7fcdd009da60 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 0x7fcdd01c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): picked mon.noname-a con 0x7fcdd00a1af0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcdd0134560 0x7fcdd01c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): picked mon.noname-c con 0x7fcdd0134560 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd009bdd0 con 0x7fcdd00a1af0 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd009bc50 con 0x7fcdd009da60 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd009bf50 con 0x7fcdd0134560 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdd5cd1640 10 monclient(hunting): authenticate will time out at 944.129155s 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 0x7fcdd01c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 0x7fcdd01c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35820/0 (socket says 172.21.15.79:35820) 2024-04-23T16:39:49.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 -- 172.21.15.79:0/1462846421 learned_addr learned my addr 172.21.15.79:0/1462846421 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): get_auth_request con 0x7fcdd00a1af0 auth_method 0 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcdd0134560 0x7fcdd01c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): get_auth_request con 0x7fcdd0134560 auth_method 0 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdd009da60 0x7fcdd013b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcdd0134560 msgr2=0x7fcdd01c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcdd0134560 0x7fcdd01c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdd009da60 msgr2=0x7fcdd013b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdd009da60 0x7fcdd013b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 monclient: found mon.noname-a 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:39:49.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd0136990 con 0x7fcdd00a1af0 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcf7fe640 10 monclient: get_auth_request con 0x7fcdd009da60 auth_method 0 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcdd009da60 0x7fcdd013b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 msgr2=0x7fcdd01c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdceffd640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 0x7fcdd01c8100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400c490 tx=0x7fcdc4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4017070 con 0x7fcdd00a1af0 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.937+0000 7fcdb7fff640 10 monclient: dump: 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.955 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc4003820 con 0x7fcdd00a1af0 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4003c10 con 0x7fcdd00a1af0 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: dump: 2024-04-23T16:39:49.956 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 5 monclient: authenticate success, global_id 4371 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 10 monclient: _renew_subs 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcdffb640 4 set_mon_vals no callback set 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd013c3b0 con 0x7fcdd00a1af0 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 10 monclient: _renew_subs 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd0138ac0 con 0x7fcdd00a1af0 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 0x7fcdd01cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd00692e0 con 0x7fcdd013bde0 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 0x7fcdd01cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient: get_auth_request con 0x7fcdd013bde0 auth_method 0 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fcdc401a360 con 0x7fcdd00a1af0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdd006ba20 con 0x7fcdd013bde0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 --2- 172.21.15.79:0/1462846421 >> v2:172.21.15.204:6812/722020384 conn(0x7fcdbc0363e0 0x7fcdbc0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (secure 0 0 0) 0x7fcdc4048430 con 0x7fcdd00a1af0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcffff640 1 --2- 172.21.15.79:0/1462846421 >> v2:172.21.15.204:6812/722020384 conn(0x7fcdbc0363e0 0x7fcdbc0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcffff640 10 monclient: get_auth_request con 0x7fcdbc0363e0 auth_method 0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 msgr2=0x7fcdd01cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 0x7fcdd01cbdc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00ad000 tx=0x7fcdc001aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00ba020 con 0x7fcdd013bde0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: dump: 2024-04-23T16:39:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcdc00abb20 con 0x7fcdd013bde0 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 msgr2=0x7fcdd01cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 0x7fcdd01cbdc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00ad000 tx=0x7fcdc001aa00 comp rx=0 tx=0).stop 2024-04-23T16:39:49.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 msgr2=0x7fcdd01cbdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:49.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.941+0000 7fcdb7fff640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd013bde0 0x7fcdd01cbdc0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00ad000 tx=0x7fcdc001aa00 comp rx=0 tx=0).stop 2024-04-23T16:39:49.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.945+0000 7fcdcffff640 1 --2- 172.21.15.79:0/1462846421 >> v2:172.21.15.204:6812/722020384 conn(0x7fcdbc0363e0 0x7fcdbc0388a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800f400 tx=0x7fcdc80076f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdd5cd1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 0x7fcdd002c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdd5cd1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd006fe20 con 0x7fcdd002c450 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 0x7fcdd002c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient: get_auth_request con 0x7fcdd002c450 auth_method 0 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.949+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/1462846421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcdd01c8be0 con 0x7fcdd002c450 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdcf7fe640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 msgr2=0x7fcdd002c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdcf7fe640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 0x7fcdd002c830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0000c10 tx=0x7fcdc00a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00ba030 con 0x7fcdd002c450 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 10 monclient: dump: 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 <== mon.0 v2:172.21.15.79:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fcdc0003620 con 0x7fcdd002c450 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 msgr2=0x7fcdd002c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 0x7fcdd002c830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0000c10 tx=0x7fcdc00a8000 comp rx=0 tx=0).stop 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 msgr2=0x7fcdd002c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdb7fff640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd002c450 0x7fcdd002c830 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0000c10 tx=0x7fcdc00a8000 comp rx=0 tx=0).stop 2024-04-23T16:39:49.965 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "leader", 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 8, 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 24795, 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: 0, 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 19, 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:39:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:39:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:39:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 2 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 3 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:39:49.977 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 >> v2:172.21.15.204:6812/722020384 conn(0x7fcdbc0363e0 msgr2=0x7fcdbc0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/1462846421 >> v2:172.21.15.204:6812/722020384 conn(0x7fcdbc0363e0 0x7fcdbc0388a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800f400 tx=0x7fcdc80076f0 comp rx=0 tx=0).stop 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 10 monclient: shutdown 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 msgr2=0x7fcdd01c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 --2- 172.21.15.79:0/1462846421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd00a1af0 0x7fcdd01c8100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400c490 tx=0x7fcdc4001e80 comp rx=0 tx=0).stop 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdcffff640 1 -- 172.21.15.79:0/1462846421 reap_dead start 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdcffff640 1 -- 172.21.15.79:0/1462846421 reap_dead start 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 shutdown_connections 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 >> 172.21.15.79:0/1462846421 conn(0x7fcdd00a6250 msgr2=0x7fcdd01324a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 shutdown_connections 2024-04-23T16:39:49.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:49.953+0000 7fcdd5cd1640 1 -- 172.21.15.79:0/1462846421 wait complete. 2024-04-23T16:39:49.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-23T16:39:49.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-23T16:39:49.980 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-23T16:39:50.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.077+0000 7f4f9cc6a640 10 monclient: get_monmap_and_config 2024-04-23T16:39:50.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.077+0000 7f4f9cc6a640 10 monclient: build_initial_monmap 2024-04-23T16:39:50.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.077+0000 7f4f9cc6a640 10 monclient: monmap: 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:50.085524+0000 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:50.085524+0000 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:50.084 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 Processor -- start 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 -- start start 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient: init 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f4f980a3600 0x7f4f980a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): picked mon.noname-a con 0x7f4f980a3600 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:50.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f98137c90 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f980a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): picked mon.noname-b con 0x7f4f98138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f980a5d40 con 0x7f4f98137c90 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f4f980a0f10 con 0x7f4f980a3600 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980a3f20 con 0x7f4f98138090 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9cc6a640 10 monclient(hunting): authenticate will time out at 944.273310s 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f980a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f4f980a3600 0x7f4f980a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f980a4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60022/0 (socket says 172.21.15.79:60022) 2024-04-23T16:39:50.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 -- 172.21.15.79:0/3636070467 learned_addr learned my addr 172.21.15.79:0/3636070467 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): get_auth_request con 0x7f4f98138090 auth_method 0 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): get_auth_request con 0x7f4f980a3600 auth_method 0 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 -- 172.21.15.79:0/3636070467 >> v2:172.21.15.79:3300/0 conn(0x7f4f980a3600 msgr2=0x7f4f980a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 --2- 172.21.15.79:0/3636070467 >> v2:172.21.15.79:3300/0 conn(0x7f4f980a3600 0x7f4f980a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 -- 172.21.15.79:0/3636070467 >> v1:172.21.15.204:6789/0 conn(0x7f4f98137c90 legacy=0x7f4f980a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 monclient: found mon.noname-b 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f96d9d640 1 -- 172.21.15.79:0/3636070467 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f980a1600 con 0x7f4f98138090 2024-04-23T16:39:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.081+0000 7f4f9659c640 1 --2- 172.21.15.79:0/3636070467 >> v2:172.21.15.79:3300/0 conn(0x7f4f980a3600 0x7f4f980a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f96d9d640 1 -- 172.21.15.79:0/3636070467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 msgr2=0x7f4f980a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f96d9d640 1 --2- 172.21.15.79:0/3636070467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f980a4ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009e50 tx=0x7f4f9001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a92a1bde7271a487 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 1 -- 172.21.15.79:0/3636070467 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f900a5640 con 0x7f4f98138090 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: dump: 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.096 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 1 -- 172.21.15.79:0/3636070467 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f900a57e0 con 0x7f4f98138090 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 5 monclient: authenticate success, global_id 4310 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 1 -- 172.21.15.79:0/3636070467 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f900a5bb0 con 0x7f4f98138090 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 4 set_mon_vals no callback set 2024-04-23T16:39:50.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9559a640 10 monclient: dump: 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: shutdown 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/3636070467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 msgr2=0x7f4f980a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/3636070467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f980a4ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009e50 tx=0x7f4f9001c8b0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/3636070467 shutdown_connections 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/3636070467 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f980a4ef0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/3636070467 >> v2:172.21.15.79:3300/0 conn(0x7f4f980a3600 0x7f4f980a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/3636070467 >> 172.21.15.79:0/3636070467 conn(0x7f4f980aa3e0 msgr2=0x7f4f980a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/3636070467 shutdown_connections 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/3636070467 wait complete. 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: build_initial_monmap 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: monmap: 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:50.099 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 Processor -- start 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- start start 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: init 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f980a3600 0x7f4f9809dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient(hunting): picked mon.noname-b con 0x7f4f980a3600 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f98137c90 0x7f4f9809e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient(hunting): picked mon.noname-a con 0x7f4f98137c90 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f9809e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient(hunting): picked mon.noname-c con 0x7f4f98138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4f981422f0 con 0x7f4f98137c90 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4f98142170 con 0x7f4f980a3600 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.085+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4f98142470 con 0x7f4f98138090 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient(hunting): authenticate will time out at 944.280023s 2024-04-23T16:39:50.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f9809e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f98137c90 0x7f4f9809e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f9809e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60032/0 (socket says 172.21.15.79:60032) 2024-04-23T16:39:50.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 -- 172.21.15.79:0/970969013 learned_addr learned my addr 172.21.15.79:0/970969013 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:50.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): get_auth_request con 0x7f4f98138090 auth_method 0 2024-04-23T16:39:50.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f98137c90 0x7f4f9809e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35852/0 (socket says 172.21.15.79:35852) 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): get_auth_request con 0x7f4f98137c90 auth_method 0 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f980a3600 msgr2=0x7f4f9809dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f980a3600 0x7f4f9809dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f980a3600 0x7f4f9809dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f98137c90 msgr2=0x7f4f9809e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f98137c90 0x7f4f9809e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 monclient: found mon.noname-c 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f98139f50 con 0x7f4f98138090 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f95d9b640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f98137c90 0x7f4f9809e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 msgr2=0x7f4f9809e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f96d9d640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f9809e860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f900af040 tx=0x7f4f90019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f900bc070 con 0x7f4f98138090 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: dump: 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f900b8850 con 0x7f4f98138090 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 5 monclient: authenticate success, global_id 4313 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient: _renew_subs 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f98138aa0 con 0x7f4f98138090 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f900a5640 con 0x7f4f98138090 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 10 monclient: dump: 2024-04-23T16:39:50.106 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient: _renew_subs 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f94d99640 4 set_mon_vals no callback set 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f98139c70 con 0x7f4f98138090 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 0x7f4f981cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4f98068ae0 con 0x7f4f981cf790 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 0x7f4f981cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient: get_auth_request con 0x7f4f981cf790 auth_method 0 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f4f900b8c20 con 0x7f4f98138090 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 1 --2- 172.21.15.79:0/970969013 >> v2:172.21.15.204:6812/722020384 conn(0x7f4f780363e0 0x7f4f780388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-04-23T16:39:50.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f9659c640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f980a0a60 con 0x7f4f981cf790 2024-04-23T16:39:50.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.089+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (secure 0 0 0) 0x7f4f900e3930 con 0x7f4f98138090 2024-04-23T16:39:50.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f95d9b640 1 --2- 172.21.15.79:0/970969013 >> v2:172.21.15.204:6812/722020384 conn(0x7f4f780363e0 0x7f4f780388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f95d9b640 10 monclient: get_auth_request con 0x7f4f780363e0 auth_method 0 2024-04-23T16:39:50.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f9659c640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 msgr2=0x7f4f981cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f9659c640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 0x7f4f981cfb70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f880107c0 tx=0x7f4f88010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8800f040 con 0x7f4f981cf790 2024-04-23T16:39:50.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 10 monclient: dump: 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.113 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4f88010e90 con 0x7f4f981cf790 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 msgr2=0x7f4f981cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 0x7f4f981cfb70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f880107c0 tx=0x7f4f88010c80 comp rx=0 tx=0).stop 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 msgr2=0x7f4f981cfb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f86ffd640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981cf790 0x7f4f981cfb70 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f880107c0 tx=0x7f4f88010c80 comp rx=0 tx=0).stop 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.093+0000 7f4f95d9b640 1 --2- 172.21.15.79:0/970969013 >> v2:172.21.15.204:6812/722020384 conn(0x7f4f780363e0 0x7f4f780388a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f9809f5f0 tx=0x7f4f8c007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 0x7f4f98134b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980a40e0 con 0x7f4f981d00b0 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 0x7f4f98134b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient: get_auth_request con 0x7f4f981d00b0 auth_method 0 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 1 -- 172.21.15.79:0/970969013 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f98072040 con 0x7f4f981d00b0 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 msgr2=0x7f4f98134b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9659c640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 0x7f4f98134b40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009fd0 tx=0x7f4f880046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88021570 con 0x7f4f981d00b0 2024-04-23T16:39:50.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 10 monclient: dump: 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 <== mon.2 v2:172.21.15.79:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f4f88029420 con 0x7f4f981d00b0 2024-04-23T16:39:50.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 msgr2=0x7f4f98134b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 0x7f4f98134b40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009fd0 tx=0x7f4f880046d0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 msgr2=0x7f4f98134b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f86ffd640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f981d00b0 0x7f4f98134b40 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009fd0 tx=0x7f4f880046d0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "peon", 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 8, 2024-04-23T16:39:50.117 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 24939, 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: 0, 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 19, 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:50.118 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:50.119 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:39:50.120 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:50.121 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:50.122 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:39:50.123 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:50.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:39:50.126 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:50.128 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.129 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 2 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 2 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.130 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.131 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:39:50.131 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 >> v2:172.21.15.204:6812/722020384 conn(0x7f4f780363e0 msgr2=0x7f4f780388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/970969013 >> v2:172.21.15.204:6812/722020384 conn(0x7f4f780363e0 0x7f4f780388a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f9809f5f0 tx=0x7f4f8c007e40 comp rx=0 tx=0).stop 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 10 monclient: shutdown 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 msgr2=0x7f4f9809e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f96d9d640 1 -- 172.21.15.79:0/970969013 reap_dead start 2024-04-23T16:39:50.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f9809e860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f900af040 tx=0x7f4f90019650 comp rx=0 tx=0).stop 2024-04-23T16:39:50.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 shutdown_connections 2024-04-23T16:39:50.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 --2- 172.21.15.79:0/970969013 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f4f98138090 0x7f4f9809e860 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 >> 172.21.15.79:0/970969013 conn(0x7f4f980aa3e0 msgr2=0x7f4f980a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:50.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 shutdown_connections 2024-04-23T16:39:50.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.101+0000 7f4f9cc6a640 1 -- 172.21.15.79:0/970969013 wait complete. 2024-04-23T16:39:50.133 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-23T16:39:50.133 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-23T16:39:50.133 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-23T16:39:50.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.237+0000 7feb8f088640 10 monclient: get_monmap_and_config 2024-04-23T16:39:50.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.237+0000 7feb8f088640 10 monclient: build_initial_monmap 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.237+0000 7feb8f088640 10 monclient: monmap: 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:50.245416+0000 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:50.245416+0000 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.244 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:50.245 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:50.245 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:50.245 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:50.245 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:50.245 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 Processor -- start 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 -- start start 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient: init 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb881355c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7feb8809c690 0x7feb8813a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): picked mon.noname-c con 0x7feb8809c690 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:50.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 0x7feb8813ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): picked mon.noname-b con 0x7feb8813aa80 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb8809bdc0 con 0x7feb881355c0 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7feb88072bb0 con 0x7feb8809c690 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7feb8809f6c0 con 0x7feb8813aa80 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8f088640 10 monclient(hunting): authenticate will time out at 944.433221s 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 0x7feb8813ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 0x7feb8813ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60062/0 (socket says 172.21.15.79:60062) 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 -- 172.21.15.79:0/3514479302 learned_addr learned my addr 172.21.15.79:0/3514479302 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): get_auth_request con 0x7feb8813aa80 auth_method 0 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-04-23T16:39:50.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 -- 172.21.15.79:0/3514479302 >> v2:172.21.15.204:3300/0 conn(0x7feb8809c690 msgr2=0x7feb8813a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb87fff640 1 --2- 172.21.15.79:0/3514479302 >> v2:172.21.15.204:3300/0 conn(0x7feb8809c690 0x7feb8813a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 --2- 172.21.15.79:0/3514479302 >> v2:172.21.15.204:3300/0 conn(0x7feb8809c690 0x7feb8813a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 -- 172.21.15.79:0/3514479302 >> v1:172.21.15.204:6789/0 conn(0x7feb881355c0 legacy=0x7feb881359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 monclient: found mon.noname-b 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.241+0000 7feb8d625640 1 -- 172.21.15.79:0/3514479302 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8809f1e0 con 0x7feb8813aa80 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8d625640 1 -- 172.21.15.79:0/3514479302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 msgr2=0x7feb8813ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8d625640 1 --2- 172.21.15.79:0/3514479302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 0x7feb8813ce70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb880a3860 tx=0x7feb8001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82c15e3c3f204302 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 1 -- 172.21.15.79:0/3514479302 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb800a5600 con 0x7feb8813aa80 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: dump: 2024-04-23T16:39:50.258 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 5 monclient: authenticate success, global_id 4322 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 1 -- 172.21.15.79:0/3514479302 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb800a57a0 con 0x7feb8813aa80 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 4 set_mon_vals no callback set 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 1 -- 172.21.15.79:0/3514479302 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb800a5b70 con 0x7feb8813aa80 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb877fe640 10 monclient: dump: 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.260 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 10 monclient: shutdown 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 -- 172.21.15.79:0/3514479302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 msgr2=0x7feb8813ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 --2- 172.21.15.79:0/3514479302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 0x7feb8813ce70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb880a3860 tx=0x7feb8001c870 comp rx=0 tx=0).stop 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 -- 172.21.15.79:0/3514479302 shutdown_connections 2024-04-23T16:39:50.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 --2- 172.21.15.79:0/3514479302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb8813aa80 0x7feb8813ce70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 --2- 172.21.15.79:0/3514479302 >> v2:172.21.15.204:3300/0 conn(0x7feb8809c690 0x7feb8813a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 -- 172.21.15.79:0/3514479302 >> 172.21.15.79:0/3514479302 conn(0x7feb880a6250 msgr2=0x7feb880a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 -- 172.21.15.79:0/3514479302 shutdown_connections 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 -- 172.21.15.79:0/3514479302 wait complete. 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 10 monclient: build_initial_monmap 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 10 monclient: monmap: 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 Processor -- start 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 1 -- start start 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.245+0000 7feb8f088640 10 monclient: init 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8809c690 0x7feb8811db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): picked mon.noname-b con 0x7feb8809c690 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 0x7feb881ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): picked mon.noname-c con 0x7feb881355c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feb8813aa80 0x7feb881ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): picked mon.noname-a con 0x7feb8813aa80 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7feb8813fd50 con 0x7feb8813aa80 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feb8813fbd0 con 0x7feb8809c690 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7feb8813fed0 con 0x7feb881355c0 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): authenticate will time out at 944.440101s 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 0x7feb881ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feb8813aa80 0x7feb881ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feb8813aa80 0x7feb881ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35858/0 (socket says 172.21.15.79:35858) 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 0x7feb881ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60066/0 (socket says 172.21.15.79:60066) 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 1 -- 172.21.15.79:0/147466668 learned_addr learned my addr 172.21.15.79:0/147466668 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:50.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): get_auth_request con 0x7feb8813aa80 auth_method 0 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): get_auth_request con 0x7feb881355c0 auth_method 0 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8ce24640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8809c690 0x7feb8811db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8809c690 msgr2=0x7feb8811db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8809c690 0x7feb8811db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feb8813aa80 msgr2=0x7feb881ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feb8813aa80 0x7feb881ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 monclient: found mon.noname-c 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8811e630 con 0x7feb881355c0 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8d625640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feb8813aa80 0x7feb881ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:39:50.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 msgr2=0x7feb881ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb87fff640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 0x7feb881ca020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c4c0 tx=0x7feb7c00c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c015070 con 0x7feb881355c0 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: dump: 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.267 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7c004220 con 0x7feb881355c0 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c011810 con 0x7feb881355c0 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 5 monclient: authenticate success, global_id 4325 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient: _renew_subs 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb8811e910 con 0x7feb881355c0 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb86ffd640 4 set_mon_vals no callback set 2024-04-23T16:39:50.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 10 monclient: dump: 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient: _renew_subs 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb88134b80 con 0x7feb881355c0 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 0x7feb881cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feb88067f20 con 0x7feb881cac30 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.249+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7feb7c0119f0 con 0x7feb881355c0 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb857fa640 1 --2- 172.21.15.79:0/147466668 >> v2:172.21.15.204:6812/722020384 conn(0x7feb740363e0 0x7feb740388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (secure 0 0 0) 0x7feb7c048ee0 con 0x7feb881355c0 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 0x7feb881cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient: get_auth_request con 0x7feb881cac30 auth_method 0 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8d625640 1 --2- 172.21.15.79:0/147466668 >> v2:172.21.15.204:6812/722020384 conn(0x7feb740363e0 0x7feb740388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8d625640 10 monclient: get_auth_request con 0x7feb740363e0 auth_method 0 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-04-23T16:39:50.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb880692e0 con 0x7feb881cac30 2024-04-23T16:39:50.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 msgr2=0x7feb881cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8ce24640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 0x7feb881cdce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d410 tx=0x7feb7800d8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb8d625640 1 --2- 172.21.15.79:0/147466668 >> v2:172.21.15.204:6812/722020384 conn(0x7feb740363e0 0x7feb740388a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feb8001af20 tx=0x7feb800a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb78016070 con 0x7feb881cac30 2024-04-23T16:39:50.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:50.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.253+0000 7feb857fa640 10 monclient: dump: 2024-04-23T16:39:50.276 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.276 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.276 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.276 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.276 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.276 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.1 v2:172.21.15.204:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb78012a10 con 0x7feb881cac30 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:50.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 msgr2=0x7feb881cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 0x7feb881cdce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d410 tx=0x7feb7800d8d0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 msgr2=0x7feb881cdce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:50.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.257+0000 7feb857fa640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb881cac30 0x7feb881cdce0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d410 tx=0x7feb7800d8d0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8f088640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:39:50.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8f088640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 0x7feb8802c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8f088640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feb88139a30 con 0x7feb8802c450 2024-04-23T16:39:50.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 0x7feb8802c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient: get_auth_request con 0x7feb8802c450 auth_method 0 2024-04-23T16:39:50.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-04-23T16:39:50.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.261+0000 7feb8ce24640 1 -- 172.21.15.79:0/147466668 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb880a1780 con 0x7feb8802c450 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8ce24640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 msgr2=0x7feb8802c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8ce24640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 0x7feb8802c860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d410 tx=0x7feb78003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb78019770 con 0x7feb8802c450 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 10 monclient: dump: 2024-04-23T16:39:50.281 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.282 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.282 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.282 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.282 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.282 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.282 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 <== mon.1 v2:172.21.15.204:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7feb78012e40 con 0x7feb8802c450 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:50.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 msgr2=0x7feb8802c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 0x7feb8802c860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d410 tx=0x7feb78003700 comp rx=0 tx=0).stop 2024-04-23T16:39:50.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 msgr2=0x7feb8802c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:39:50.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb857fa640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb8802c450 0x7feb8802c860 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb7800d410 tx=0x7feb78003700 comp rx=0 tx=0).stop 2024-04-23T16:39:50.285 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:39:50.285 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:39:50.285 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:39:50.285 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "peon", 2024-04-23T16:39:50.285 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 8, 2024-04-23T16:39:50.286 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 25071, 2024-04-23T16:39:50.286 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:39:50.286 INFO:teuthology.orchestra.run.smithi079.stdout: 0, 2024-04-23T16:39:50.286 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:39:50.286 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:39:50.286 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.287 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 19, 2024-04-23T16:39:50.287 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:39:50.287 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:39:50.287 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:39:50.287 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:50.287 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:50.288 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:50.288 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:50.288 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:50.288 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:50.288 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:50.288 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:39:50.289 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:50.290 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:50.291 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:50.291 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:50.291 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:50.291 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:50.291 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.291 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.292 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:39:50.292 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:39:50.292 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:39:50.292 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:39:50.292 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:39:50.292 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:39:50.293 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:39:50.293 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:39:50.293 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:39:50.293 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:39:50.293 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:39:50.294 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:39:50.294 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:39:50.294 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:39:50.294 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:39:50.294 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:39:50.294 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:39:50.295 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:39:50.295 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:39:50.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:39:50.295 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:39:50.295 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:39:50.295 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:39:50.296 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:39:50.296 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:39:50.296 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:39:50.296 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:39:50.296 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:39:50.296 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.297 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:39:50.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:39:50.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.297 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:39:50.297 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:39:50.298 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:50.298 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:50.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.298 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:50.298 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.300 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.300 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:39:50.300 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:39:50.300 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:50.301 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:50.301 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:50.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.301 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:39:50.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:39:50.302 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:50.302 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:50.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.302 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:50.302 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:39:50.302 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.303 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.303 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:50.303 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:39:50.303 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.303 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.304 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.304 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:39:50.304 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:39:50.304 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:50.304 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:50.305 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:50.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.305 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:39:50.305 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:39:50.305 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:39:50.306 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:39:50.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.306 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:39:50.307 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:39:50.307 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.307 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.307 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:39:50.307 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:39:50.308 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:39:50.308 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.308 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.308 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:39:50.308 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.309 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:39:50.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 2 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.311 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 6 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": [ 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:39:50.312 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:39:50.313 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:39:50.313 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:39:50.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:39:50.313 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:39:50.313 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:39:50.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 >> v2:172.21.15.204:6812/722020384 conn(0x7feb740363e0 msgr2=0x7feb740388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 --2- 172.21.15.79:0/147466668 >> v2:172.21.15.204:6812/722020384 conn(0x7feb740363e0 0x7feb740388a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feb8001af20 tx=0x7feb800a7070 comp rx=0 tx=0).stop 2024-04-23T16:39:50.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 10 monclient: shutdown 2024-04-23T16:39:50.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:50.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 msgr2=0x7feb881ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 --2- 172.21.15.79:0/147466668 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7feb881355c0 0x7feb881ca020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c4c0 tx=0x7feb7c00c980 comp rx=0 tx=0).stop 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8d625640 1 -- 172.21.15.79:0/147466668 reap_dead start 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8d625640 1 -- 172.21.15.79:0/147466668 reap_dead start 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 shutdown_connections 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 >> 172.21.15.79:0/147466668 conn(0x7feb880a6250 msgr2=0x7feb880a1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 shutdown_connections 2024-04-23T16:39:50.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.265+0000 7feb8f088640 1 -- 172.21.15.79:0/147466668 wait complete. 2024-04-23T16:39:50.316 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-23T16:39:50.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-23T16:39:50.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-23T16:39:50.317 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-23T16:39:50.317 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-23T16:39:50.317 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-23T16:39:50.317 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:39:50.328 INFO:tasks.ceph.mon.a:Stopped 2024-04-23T16:39:50.328 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-23T16:39:50.328 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:39:50.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient: get_monmap_and_config 2024-04-23T16:39:50.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient: build_initial_monmap 2024-04-23T16:39:50.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient: monmap: 2024-04-23T16:39:50.484 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:50.486497+0000 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:50.486497+0000 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:50.485 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 1 Processor -- start 2024-04-23T16:39:50.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 1 -- start start 2024-04-23T16:39:50.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient: init 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2ac06ca30 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fa2ac0a3a80 0x7fa2ac0a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient(hunting): picked mon.noname-a con 0x7fa2ac0a3a80 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2ac13c8c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac0a2f20 con 0x7fa2ac06ca30 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac0a5860 con 0x7fa2ac13c8c0 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fa2ac0a1b50 con 0x7fa2ac0a3a80 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2b09fe640 10 monclient(hunting): authenticate will time out at 944.674351s 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2aa59c640 1 -- >> v1:172.21.15.79:6789/0 conn(0x7fa2ac06ca30 legacy=0x7fa2ac0a3370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2a9d9b640 1 -- >> v2:172.21.15.79:3300/0 conn(0x7fa2ac0a3a80 msgr2=0x7fa2ac0a2830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:39:50.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2a9d9b640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fa2ac0a3a80 0x7fa2ac0a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:50.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2aad9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa2ac13c8c0 0x7fa2ac13ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.79:54274/0 (socket says 172.21.15.79:54274) 2024-04-23T16:39:50.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.481+0000 7fa2aad9d640 1 -- 172.21.15.79:0/3053799787 learned_addr learned my addr 172.21.15.79:0/3053799787 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:39:50.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aad9d640 1 -- 172.21.15.79:0/3053799787 >> v1:172.21.15.204:6789/0 conn(0x7fa2ac13c8c0 legacy=0x7fa2ac13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1939482150 0 0) 0x7fa2ac0a5860 con 0x7fa2ac13c8c0 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): _negotiate 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): my global_id is 4345 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa298003530 con 0x7fa2ac13c8c0 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 226597811 0 0) 0x7fa298003530 con 0x7fa2ac13c8c0 2024-04-23T16:39:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 >> v2:172.21.15.79:3300/0 conn(0x7fa2ac0a3a80 msgr2=0x7fa2ac0a2830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 --2- 172.21.15.79:0/3053799787 >> v2:172.21.15.79:3300/0 conn(0x7fa2ac0a3a80 0x7fa2ac0a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 >> v1:172.21.15.79:6789/0 conn(0x7fa2ac06ca30 legacy=0x7fa2ac0a3370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 monclient: found mon.noname-c-legacy 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.204:6789/0 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac0a1590 con 0x7fa2ac13c8c0 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1099920027 0 0) 0x7fa29c001930 con 0x7fa2ac13c8c0 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-23T16:39:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: dump: 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] but i'm connected to v1:172.21.15.204:6789/0, reconnecting 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 >> v1:172.21.15.204:6789/0 conn(0x7fa2ac13c8c0 legacy=0x7fa2ac13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa298003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): picked mon.c con 0x7fa2980039c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa298004810 0x7fa298004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): picked mon.b con 0x7fa298004810 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2980052f0 0x7fa29800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): picked mon.a con 0x7fa2980052f0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac0a1b50 con 0x7fa2980052f0 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa29800c240 con 0x7fa298004810 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fa29800c440 con 0x7fa2980039c0 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a959a640 10 monclient(hunting): _finish_auth 0 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aad9d640 1 -- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2980052f0 msgr2=0x7fa29800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aad9d640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2980052f0 0x7fa29800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa298003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): get_auth_request con 0x7fa2980039c0 auth_method 0 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:39:50.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a9d9b640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa298004810 0x7fa298004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a9d9b640 10 monclient(hunting): get_auth_request con 0x7fa298004810 auth_method 0 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2a9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_done global_id 4345 payload 293 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3053799787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa298004810 msgr2=0x7fa298004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa298004810 0x7fa298004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2980052f0 msgr2=0x7fa29800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2980052f0 0x7fa29800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 monclient: found mon.c 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3053799787 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa298003530 con 0x7fa2980039c0 2024-04-23T16:39:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2aad9d640 1 -- 172.21.15.79:0/3053799787 reap_dead start 2024-04-23T16:39:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2b09fe640 5 monclient: authenticate success, global_id 4345 2024-04-23T16:39:50.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.485+0000 7fa2b09fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 msgr2=0x7fa298003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa298003dc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4003d10 tx=0x7fa2a401cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3a643d662c8d2e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4009a30 con 0x7fa2980039c0 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: dump: 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a40a9070 con 0x7fa2980039c0 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 1 -- 172.21.15.79:0/3053799787 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a40a59f0 con 0x7fa2980039c0 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 4 set_mon_vals no callback set 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2a959a640 10 monclient: dump: 2024-04-23T16:39:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 10 monclient: shutdown 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 msgr2=0x7fa298003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa298003dc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4003d10 tx=0x7fa2a401cab0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3053799787 shutdown_connections 2024-04-23T16:39:50.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3053799787 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa298003dc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3053799787 >> 172.21.15.79:0/3053799787 conn(0x7fa2ac0aa3e0 msgr2=0x7fa2ac0a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3053799787 shutdown_connections 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.489+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3053799787 wait complete. 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: build_initial_monmap 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: monmap: 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:50.506 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 Processor -- start 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 -- start start 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: init 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa2ac1cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): picked mon.noname-c con 0x7fa2980039c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2ac1cd200 0x7fa2ac09d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): picked mon.noname-a con 0x7fa2ac1cd200 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2ac1cd640 0x7fa2ac09d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): picked mon.noname-b con 0x7fa2ac1cd640 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac0a2bc0 con 0x7fa2ac1cd200 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac0a2a40 con 0x7fa2ac1cd640 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ac06ce10 con 0x7fa2980039c0 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient(hunting): authenticate will time out at 944.684931s 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a9d9b640 1 -- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2ac1cd200 msgr2=0x7fa2ac09d200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a9d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2ac1cd200 0x7fa2ac09d200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa2ac1cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa2ac1cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60092/0 (socket says 172.21.15.79:60092) 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3588701748 learned_addr learned my addr 172.21.15.79:0/3588701748 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): get_auth_request con 0x7fa2980039c0 auth_method 0 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3588701748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2ac1cd640 msgr2=0x7fa2ac09d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2ac1cd640 0x7fa2ac09d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2ac1cd200 msgr2=0x7fa2ac09d200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2ac1cd200 0x7fa2ac09d200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 monclient: found mon.noname-c 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:50.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3588701748 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac120710 con 0x7fa2980039c0 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 -- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 msgr2=0x7fa2ac1cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2aa59c640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa2ac1cccc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2a401c8e0 tx=0x7fa298005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4009a30 con 0x7fa2980039c0 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: dump: 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.510 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a40aea90 con 0x7fa2980039c0 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a40a5e20 con 0x7fa2980039c0 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 5 monclient: authenticate success, global_id 4328 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: _renew_subs 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac0a4000 con 0x7fa2980039c0 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a8d99640 4 set_mon_vals no callback set 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2a2ffd640 10 monclient: dump: 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:50.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: _renew_subs 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.493+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac09f780 con 0x7fa2980039c0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2b09fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2b09fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2b09fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac068ae0 con 0x7fa2980039c0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fa2a40b9420 con 0x7fa2980039c0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2a2ffd640 1 --2- 172.21.15.79:0/3588701748 >> v2:172.21.15.204:6812/722020384 conn(0x7fa28c036390 0x7fa28c038850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (secure 0 0 0) 0x7fa2a40e3780 con 0x7fa2980039c0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2a9d9b640 1 --2- 172.21.15.79:0/3588701748 >> v2:172.21.15.204:6812/722020384 conn(0x7fa28c036390 0x7fa28c038850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:50.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.497+0000 7fa2a9d9b640 10 monclient: get_auth_request con 0x7fa28c036390 auth_method 0 2024-04-23T16:39:50.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.501+0000 7fa2a9d9b640 1 --2- 172.21.15.79:0/3588701748 >> v2:172.21.15.204:6812/722020384 conn(0x7fa28c036390 0x7fa28c038850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa294006fd0 tx=0x7fa294008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:50.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.501+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa2a4016a80 con 0x7fa2980039c0 2024-04-23T16:39:50.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.501+0000 7fa2a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:50.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.501+0000 7fa2a2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:50.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2b09fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:39:50.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2b09fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:50.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2b09fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:50.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa2ac0a56a0 con 0x7fa2980039c0 2024-04-23T16:39:50.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2a2ffd640 1 -- 172.21.15.79:0/3588701748 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fa2a40b8380 con 0x7fa2980039c0 2024-04-23T16:39:50.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:50.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.705+0000 7fa2a2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:50.710 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 >> v2:172.21.15.204:6812/722020384 conn(0x7fa28c036390 msgr2=0x7fa28c038850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3588701748 >> v2:172.21.15.204:6812/722020384 conn(0x7fa28c036390 0x7fa28c038850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa294006fd0 tx=0x7fa294008040 comp rx=0 tx=0).stop 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 10 monclient: shutdown 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 msgr2=0x7fa2ac1cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa2ac1cccc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2a401c8e0 tx=0x7fa298005840 comp rx=0 tx=0).stop 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 shutdown_connections 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3588701748 >> v2:172.21.15.204:6812/722020384 conn(0x7fa28c036390 0x7fa28c038850 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2ac1cd640 0x7fa2ac09d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2ac1cd200 0x7fa2ac09d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 --2- 172.21.15.79:0/3588701748 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa2980039c0 0x7fa2ac1cccc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:50.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 >> 172.21.15.79:0/3588701748 conn(0x7fa2ac0aa3e0 msgr2=0x7fa2ac0a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:50.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 shutdown_connections 2024-04-23T16:39:50.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:50.709+0000 7fa2b09fe640 1 -- 172.21.15.79:0/3588701748 wait complete. 2024-04-23T16:39:53.729 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:39:53.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient: get_monmap_and_config 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient: build_initial_monmap 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient: monmap: 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:53.858835+0000 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:53.858835+0000 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:53.858 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:53.858 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:53.858 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 Processor -- start 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 -- start start 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient: init 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bfc0a32c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 0x7f7bfc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): picked mon.noname-b con 0x7f7bfc0a36a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7bfc0a3ac0 0x7f7bfc13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:53.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): picked mon.noname-a con 0x7f7bfc0a3ac0 addr v2:172.21.15.79:3300/0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bfc138340 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f7bfc0a5be0 con 0x7f7bfc0a3ac0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7bfc0a14f0 con 0x7f7bfc0a36a0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7c00b43640 10 monclient(hunting): authenticate will time out at 948.046648s 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bfad9d640 1 -- >> v2:172.21.15.79:3300/0 conn(0x7f7bfc0a3ac0 msgr2=0x7f7bfc13ede0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bfad9d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7bfc0a3ac0 0x7f7bfc13ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 0x7f7bfc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 0x7f7bfc13c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60096/0 (socket says 172.21.15.79:60096) 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 -- 172.21.15.79:0/168071197 learned_addr learned my addr 172.21.15.79:0/168071197 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): get_auth_request con 0x7f7bfc0a36a0 auth_method 0 2024-04-23T16:39:53.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 -- 172.21.15.79:0/168071197 >> v2:172.21.15.79:3300/0 conn(0x7f7bfc0a3ac0 msgr2=0x7f7bfc13ede0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 --2- 172.21.15.79:0/168071197 >> v2:172.21.15.79:3300/0 conn(0x7f7bfc0a3ac0 0x7f7bfc13ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 -- 172.21.15.79:0/168071197 >> v1:172.21.15.204:6789/0 conn(0x7f7bfc0a32c0 legacy=0x7f7bfc137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient: _finish_hunting 0 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 monclient: found mon.noname-b 2024-04-23T16:39:53.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:53.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:53.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.853+0000 7f7bf9d9b640 1 -- 172.21.15.79:0/168071197 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bfc0a17d0 con 0x7f7bfc0a36a0 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf9d9b640 1 -- 172.21.15.79:0/168071197 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 msgr2=0x7f7bfc13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf9d9b640 1 --2- 172.21.15.79:0/168071197 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 0x7f7bfc13c890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00098a0 tx=0x7f7bf001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bec88ffdc7b5ac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 1 -- 172.21.15.79:0/168071197 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00b0070 con 0x7f7bfc0a36a0 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: dump: 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:53.865 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: _finish_auth 0 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: _check_auth_tickets 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 1 -- 172.21.15.79:0/168071197 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf00ab600 con 0x7f7bfc0a36a0 2024-04-23T16:39:53.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7c00b43640 5 monclient: authenticate success, global_id 4331 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7c00b43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7c00b43640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 1 -- 172.21.15.79:0/168071197 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0a74e0 con 0x7f7bfc0a36a0 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7c00b43640 4 set_mon_vals no callback set 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.857+0000 7f7bf959a640 10 monclient: dump: 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.867 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient: shutdown 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- 172.21.15.79:0/168071197 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 msgr2=0x7f7bfc13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 --2- 172.21.15.79:0/168071197 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 0x7f7bfc13c890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00098a0 tx=0x7f7bf001c470 comp rx=0 tx=0).stop 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- 172.21.15.79:0/168071197 shutdown_connections 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 --2- 172.21.15.79:0/168071197 >> v2:172.21.15.79:3300/0 conn(0x7f7bfc0a3ac0 0x7f7bfc13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 --2- 172.21.15.79:0/168071197 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a36a0 0x7f7bfc13c890 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- 172.21.15.79:0/168071197 >> 172.21.15.79:0/168071197 conn(0x7f7bfc0aa4c0 msgr2=0x7f7bfc0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- 172.21.15.79:0/168071197 shutdown_connections 2024-04-23T16:39:53.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- 172.21.15.79:0/168071197 wait complete. 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient: build_initial_monmap 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient: monmap: 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:53.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 Processor -- start 2024-04-23T16:39:53.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- start start 2024-04-23T16:39:53.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient: init 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 0x7f7bfc133a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): picked mon.noname-c con 0x7f7bfc0a32c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bfc0a36a0 0x7f7bfc133fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): picked mon.noname-b con 0x7f7bfc0a36a0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7bfc0a3ac0 0x7f7bfc138640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): picked mon.noname-a con 0x7f7bfc0a3ac0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:53.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7bfc141f50 con 0x7f7bfc0a3ac0 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7bfc141dd0 con 0x7f7bfc0a36a0 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7bfc1420d0 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7c00b43640 10 monclient(hunting): authenticate will time out at 948.053854s 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfad9d640 1 -- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7bfc0a3ac0 msgr2=0x7f7bfc138640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7bfc0a3ac0 0x7f7bfc138640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 0x7f7bfc133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 0x7f7bfc133a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:60104/0 (socket says 172.21.15.79:60104) 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 -- 172.21.15.79:0/2968714450 learned_addr learned my addr 172.21.15.79:0/2968714450 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): get_auth_request con 0x7f7bfc0a32c0 auth_method 0 2024-04-23T16:39:53.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bf9d9b640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bfc0a36a0 0x7f7bfc133fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 -- 172.21.15.79:0/2968714450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bfc0a36a0 msgr2=0x7f7bfc133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bfc0a36a0 0x7f7bfc133fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 -- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7bfc0a3ac0 msgr2=0x7f7bfc138640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7bfc0a3ac0 0x7f7bfc138640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient: _finish_hunting 0 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 monclient: found mon.noname-c 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:53.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.861+0000 7f7bfa59c640 1 -- 172.21.15.79:0/2968714450 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bfc0a3ff0 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7bfa59c640 1 -- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 msgr2=0x7f7bfc133a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7bfa59c640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 0x7f7bfc133a90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c450 tx=0x7f7bec00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec012a40 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: dump: 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: _finish_auth 0 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: _check_auth_tickets 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec017070 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 5 monclient: authenticate success, global_id 4334 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 monclient: _renew_subs 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bfc13c330 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec012e40 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 10 monclient: dump: 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:53.877 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 monclient: _renew_subs 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7bebfff640 4 set_mon_vals no callback set 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bfc13a920 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:53.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bfc13bba0 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f7bec01c460 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 1 --2- 172.21.15.79:0/2968714450 >> v2:172.21.15.204:6812/722020384 conn(0x7f7bdc036410 0x7f7bdc0388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (secure 0 0 0) 0x7f7bec048f90 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7bf9d9b640 1 --2- 172.21.15.79:0/2968714450 >> v2:172.21.15.204:6812/722020384 conn(0x7f7bdc036410 0x7f7bdc0388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.865+0000 7f7bf9d9b640 10 monclient: get_auth_request con 0x7f7bdc036410 auth_method 0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.869+0000 7f7bf9d9b640 1 --2- 172.21.15.79:0/2968714450 >> v2:172.21.15.204:6812/722020384 conn(0x7f7bdc036410 0x7f7bdc0388d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00ab000 tx=0x7f7bf00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.869+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7bfc0a3ff0 con 0x7f7bfc0a32c0 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.869+0000 7f7beaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:53.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:53.869+0000 7f7beaffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:54.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7bfc0692e0 con 0x7f7bfc0a32c0 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7beaffd640 1 -- 172.21.15.79:0/2968714450 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7bec01b3c0 con 0x7f7bfc0a32c0 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7beaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7beaffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:54.097 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 >> v2:172.21.15.204:6812/722020384 conn(0x7f7bdc036410 msgr2=0x7f7bdc0388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 --2- 172.21.15.79:0/2968714450 >> v2:172.21.15.204:6812/722020384 conn(0x7f7bdc036410 0x7f7bdc0388d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00ab000 tx=0x7f7bf00a6000 comp rx=0 tx=0).stop 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 10 monclient: shutdown 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 msgr2=0x7f7bfc133a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 0x7f7bfc133a90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c450 tx=0x7f7bec00c910 comp rx=0 tx=0).stop 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 shutdown_connections 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 --2- 172.21.15.79:0/2968714450 >> v2:172.21.15.204:6812/722020384 conn(0x7f7bdc036410 0x7f7bdc0388d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7bfc0a3ac0 0x7f7bfc138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bfc0a36a0 0x7f7bfc133fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 --2- 172.21.15.79:0/2968714450 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7bfc0a32c0 0x7f7bfc133a90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:54.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.093+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 >> 172.21.15.79:0/2968714450 conn(0x7f7bfc0aa4c0 msgr2=0x7f7bfc1367a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:54.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.097+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 shutdown_connections 2024-04-23T16:39:54.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:54.097+0000 7f7c00b43640 1 -- 172.21.15.79:0/2968714450 wait complete. 2024-04-23T16:39:57.117 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:39:57.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient: get_monmap_and_config 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient: build_initial_monmap 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient: monmap: 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:39:57.247926+0000 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:39:57.247926+0000 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:39:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:39:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:39:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:39:57.247 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 Processor -- start 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 -- start start 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient: init 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcbdc135a20 0x7fcbdc135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): picked mon.noname-c con 0x7fcbdc135a20 addr v2:172.21.15.204:3300/0 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc09ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc09f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:57.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbdc135300 addr v1:172.21.15.79:6789/0 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbdc072f80 con 0x7fcbdc135300 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcbdc09fb10 con 0x7fcbdc135a20 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcbdc09fdf0 con 0x7fcbdc09f430 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbe1ef6640 10 monclient(hunting): authenticate will time out at 951.435776s 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.241+0000 7fcbdbfff640 1 -- >> v1:172.21.15.79:6789/0 conn(0x7fcbdc135300 legacy=0x7fcbdc09bf40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc09ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc09ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:49944/0 (socket says 172.21.15.79:49944) 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 -- 172.21.15.79:0/2107915686 learned_addr learned my addr 172.21.15.79:0/2107915686 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): get_auth_request con 0x7fcbdc09f430 auth_method 0 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:57.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 1 --2- 172.21.15.79:0/2107915686 >> v2:172.21.15.204:3300/0 conn(0x7fcbdc135a20 0x7fcbdc135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): get_auth_request con 0x7fcbdc135a20 auth_method 0 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 -- 172.21.15.79:0/2107915686 >> v2:172.21.15.204:3300/0 conn(0x7fcbdc135a20 msgr2=0x7fcbdc135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 --2- 172.21.15.79:0/2107915686 >> v2:172.21.15.204:3300/0 conn(0x7fcbdc135a20 0x7fcbdc135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 -- 172.21.15.79:0/2107915686 >> v1:172.21.15.79:6789/0 conn(0x7fcbdc135300 legacy=0x7fcbdc09bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:39:57.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient: _finish_hunting 0 2024-04-23T16:39:57.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 monclient: found mon.noname-b 2024-04-23T16:39:57.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:57.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:39:57.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.245+0000 7fcbdaffd640 1 -- 172.21.15.79:0/2107915686 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc06de30 con 0x7fcbdc09f430 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbdaffd640 1 -- 172.21.15.79:0/2107915686 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 msgr2=0x7fcbdc09ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbdaffd640 1 --2- 172.21.15.79:0/2107915686 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc09ba00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009870 tx=0x7fcbcc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32f234c5ab050925 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 1 -- 172.21.15.79:0/2107915686 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0b0070 con 0x7fcbdc09f430 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: dump: 2024-04-23T16:39:57.253 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:57.254 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: _finish_auth 0 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: _check_auth_tickets 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 1 -- 172.21.15.79:0/2107915686 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0ab600 con 0x7fcbdc09f430 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 1 -- 172.21.15.79:0/2107915686 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0ab9d0 con 0x7fcbdc09f430 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 5 monclient: authenticate success, global_id 4337 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 10 monclient: get_monmap_and_config success 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 4 set_mon_vals no callback set 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbda7fc640 10 monclient: dump: 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:57.255 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:57.256 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:57.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 10 monclient: shutdown 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/2107915686 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 msgr2=0x7fcbdc09ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/2107915686 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc09ba00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009870 tx=0x7fcbcc01c3e0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/2107915686 shutdown_connections 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/2107915686 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc09ba00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/2107915686 >> v2:172.21.15.204:3300/0 conn(0x7fcbdc135a20 0x7fcbdc135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/2107915686 >> 172.21.15.79:0/2107915686 conn(0x7fcbdc0a25b0 msgr2=0x7fcbdc0a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/2107915686 shutdown_connections 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/2107915686 wait complete. 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 10 monclient: build_initial_monmap 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 10 monclient: monmap: 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:39:57.261 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 Processor -- start 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 1 -- start start 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.249+0000 7fcbe1ef6640 10 monclient: init 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _reopen_session rank -1 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc13c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:57.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): picked mon.noname-c con 0x7fcbdc09f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbdc135300 0x7fcbdc134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc135300 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcbdc135a20 0x7fcbdc1cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): picked mon.noname-a con 0x7fcbdc135a20 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc1400b0 con 0x7fcbdc135a20 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc09c860 con 0x7fcbdc135300 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcbdc140230 con 0x7fcbdc09f430 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): _renew_subs 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient(hunting): authenticate will time out at 951.444147s 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdbfff640 1 -- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcbdc135a20 msgr2=0x7fcbdc1cd550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc13c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:57.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcbdc135a20 0x7fcbdc1cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc13c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:49960/0 (socket says 172.21.15.79:49960) 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 -- 172.21.15.79:0/1601839532 learned_addr learned my addr 172.21.15.79:0/1601839532 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): get_auth_request con 0x7fcbdc09f430 auth_method 0 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 -- 172.21.15.79:0/1601839532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbdc135300 msgr2=0x7fcbdc134520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:57.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbdc135300 0x7fcbdc134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 -- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcbdc135a20 msgr2=0x7fcbdc1cd550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcbdc135a20 0x7fcbdc1cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 monclient: found mon.noname-c 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 -- 172.21.15.79:0/1601839532 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc13cfe0 con 0x7fcbdc09f430 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 -- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 msgr2=0x7fcbdc13c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbdb7fe640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc13c3a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400e930 tx=0x7fcbc400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4017070 con 0x7fcbdc09f430 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:57.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: dump: 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: _finish_auth 0 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: _check_auth_tickets 2024-04-23T16:39:57.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc400cb90 con 0x7fcbdc09f430 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 5 monclient: authenticate success, global_id 4340 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _renew_subs 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc11e2c0 con 0x7fcbdc09f430 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4013470 con 0x7fcbdc09f430 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 10 monclient: dump: 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:39:57.267 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd9ffb640 4 set_mon_vals no callback set 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _renew_subs 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc11d0a0 con 0x7fcbdc09f430 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbdc0692e0 con 0x7fcbdc09f430 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.253+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fcbc401a070 con 0x7fcbdc09f430 2024-04-23T16:39:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.257+0000 7fcbd3fff640 1 --2- 172.21.15.79:0/1601839532 >> v2:172.21.15.204:6812/722020384 conn(0x7fcbbc0363e0 0x7fcbbc0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.257+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3898+0+0 (secure 0 0 0) 0x7fcbc40491e0 con 0x7fcbdc09f430 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.257+0000 7fcbdaffd640 1 --2- 172.21.15.79:0/1601839532 >> v2:172.21.15.204:6812/722020384 conn(0x7fcbbc0363e0 0x7fcbbc0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.257+0000 7fcbdaffd640 10 monclient: get_auth_request con 0x7fcbbc0363e0 auth_method 0 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.257+0000 7fcbdaffd640 1 --2- 172.21.15.79:0/1601839532 >> v2:172.21.15.204:6812/722020384 conn(0x7fcbbc0363e0 0x7fcbbc0388a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0ab000 tx=0x7fcbcc004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.261+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcbc4013650 con 0x7fcbdc09f430 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.261+0000 7fcbd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:39:57.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.261+0000 7fcbd3fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:39:57.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:39:57.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:57.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:39:57.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbdc071a20 con 0x7fcbdc09f430 2024-04-23T16:39:57.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbd3fff640 1 -- 172.21.15.79:0/1601839532 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fcbc4047f50 con 0x7fcbdc09f430 2024-04-23T16:39:57.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:39:57.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbd3fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:39:57.493 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 >> v2:172.21.15.204:6812/722020384 conn(0x7fcbbc0363e0 msgr2=0x7fcbbc0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/1601839532 >> v2:172.21.15.204:6812/722020384 conn(0x7fcbbc0363e0 0x7fcbbc0388a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0ab000 tx=0x7fcbcc004740 comp rx=0 tx=0).stop 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 10 monclient: shutdown 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 msgr2=0x7fcbdc13c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc13c3a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400e930 tx=0x7fcbc400edf0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 shutdown_connections 2024-04-23T16:39:57.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/1601839532 >> v2:172.21.15.204:6812/722020384 conn(0x7fcbbc0363e0 0x7fcbbc0388a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcbdc135a20 0x7fcbdc1cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbdc135300 0x7fcbdc134520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 --2- 172.21.15.79:0/1601839532 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcbdc09f430 0x7fcbdc13c3a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:39:57.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 >> 172.21.15.79:0/1601839532 conn(0x7fcbdc0a25b0 msgr2=0x7fcbdc136400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:39:57.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 shutdown_connections 2024-04-23T16:39:57.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:39:57.489+0000 7fcbe1ef6640 1 -- 172.21.15.79:0/1601839532 wait complete. 2024-04-23T16:40:00.512 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:40:00.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient: get_monmap_and_config 2024-04-23T16:40:00.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient: build_initial_monmap 2024-04-23T16:40:00.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient: monmap: 2024-04-23T16:40:00.640 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:40:00.640 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:40:00.642575+0000 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:40:00.642575+0000 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:40:00.641 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 Processor -- start 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 -- start start 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient: init 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 0x7fa71c135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): picked mon.noname-a con 0x7fa71c135a20 addr v2:172.21.15.79:3300/0 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa71c09f430 addr v1:172.21.15.204:6789/0 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:00.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): picked mon.noname-b con 0x7fa71c135300 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa71c072f80 con 0x7fa71c09f430 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fa71c09fb10 con 0x7fa71c135a20 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fa71c09fdf0 con 0x7fa71c135300 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71affd640 1 -- >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 msgr2=0x7fa71c135e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa72184d640 10 monclient(hunting): authenticate will time out at 954.830667s 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71affd640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 0x7fa71c135e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:49964/0 (socket says 172.21.15.79:49964) 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 learned_addr learned my addr 172.21.15.79:0/4190274264 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request con 0x7fa71c135300 auth_method 0 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:00.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:00.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:00.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T16:40:00.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:00.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:00.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.637+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:00.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.641+0000 7fa71a7fc640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.204:6789/0 conn(0x7fa71c09f430 legacy=0x7fa71c09ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 msgr2=0x7fa71c135e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 0x7fa71c135e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request con 0x7fa71c135300 auth_method 0 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:00.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:00.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:00.841+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T16:40:01.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 msgr2=0x7fa71c135e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:01.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 0x7fa71c135e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request con 0x7fa71c135300 auth_method 0 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:01.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:01.245+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 msgr2=0x7fa71c135e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 0x7fa71c135e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request con 0x7fa71c135300 auth_method 0 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:02.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:02.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T16:40:02.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:02.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:02.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:02.049+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T16:40:03.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): tick 2024-04-23T16:40:03.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-23T16:40:03.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 monclient(hunting): continuing hunt 2024-04-23T16:40:03.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-23T16:40:03.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 msgr2=0x7fa71c09c110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:40:03.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa71c135300 0x7fa71c09c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:03.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 msgr2=0x7fa71c135e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:40:03.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.79:3300/0 conn(0x7fa71c135a20 0x7fa71c135e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:03.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.204:6789/0 conn(0x7fa71c09f430 legacy=0x7fa71c09ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:03.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient: _add_conns ranks=[4,0] 2024-04-23T16:40:03.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:03.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fa6fc003770 addr v2:172.21.15.204:3300/0 2024-04-23T16:40:03.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6fc0062c0 addr v1:172.21.15.79:6789/0 2024-04-23T16:40:03.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 -- 172.21.15.79:0/4190274264 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa71c06de30 con 0x7fa6fc0062c0 2024-04-23T16:40:03.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:03.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 1 -- 172.21.15.79:0/4190274264 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa71c09fb10 con 0x7fa6fc003770 2024-04-23T16:40:03.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa718ff9640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:03.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71a7fc640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.79:6789/0 conn(0x7fa6fc0062c0 legacy=0x7fa6fc008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 10 monclient(hunting): get_auth_request con 0x7fa6fc003770 auth_method 0 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:40:03.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:03.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:03.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.637+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:03.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71a7fc640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.79:6789/0 conn(0x7fa6fc0062c0 legacy=0x7fa6fc008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 10 monclient(hunting): get_auth_request con 0x7fa6fc003770 auth_method 0 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:03.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:03.841+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T16:40:04.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71a7fc640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.79:6789/0 conn(0x7fa6fc0062c0 legacy=0x7fa6fc008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:40:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 10 monclient(hunting): get_auth_request con 0x7fa6fc003770 auth_method 0 2024-04-23T16:40:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-23T16:40:04.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:40:04.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:40:04.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:04.245+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T16:40:05.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71a7fc640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.79:6789/0 conn(0x7fa6fc0062c0 legacy=0x7fa6fc008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:40:05.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> v2:172.21.15.204:3300/0 conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): get_auth_request con 0x7fa6fc003770 auth_method 0 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> v1:172.21.15.79:6789/0 conn(0x7fa6fc0062c0 legacy=0x7fa6fc008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 1 monclient: found mon.noname-c 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:40:05.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.049+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6fc00b070 con 0x7fa6fc003770 2024-04-23T16:40:05.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa71affd640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa71affd640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6fc003770 0x7fa6fc005bd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa714003990 tx=0x7fa71401d180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5619d9fe1e3377e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 1 -- 172.21.15.79:0/4190274264 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7140b3070 con 0x7fa6fc003770 2024-04-23T16:40:05.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: dump: 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.063 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 1 -- 172.21.15.79:0/4190274264 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7140ae410 con 0x7fa6fc003770 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa72184d640 5 monclient: authenticate success, global_id 4351 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 1 -- 172.21.15.79:0/4190274264 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7140ae800 con 0x7fa6fc003770 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa719ffb640 10 monclient: dump: 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.064 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa72184d640 10 monclient: get_monmap_and_config success 2024-04-23T16:40:05.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.057+0000 7fa72184d640 4 set_mon_vals no callback set 2024-04-23T16:40:05.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 10 monclient: shutdown 2024-04-23T16:40:05.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:40:05.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 -- 172.21.15.79:0/4190274264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6fc003770 msgr2=0x7fa6fc005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6fc003770 0x7fa6fc005bd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa714003990 tx=0x7fa71401d180 comp rx=0 tx=0).stop 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa71b7fe640 1 -- 172.21.15.79:0/4190274264 reap_dead start 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 -- 172.21.15.79:0/4190274264 shutdown_connections 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 --2- 172.21.15.79:0/4190274264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6fc003770 0x7fa6fc005bd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 -- 172.21.15.79:0/4190274264 >> 172.21.15.79:0/4190274264 conn(0x7fa71c0a25b0 msgr2=0x7fa71c0a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 -- 172.21.15.79:0/4190274264 shutdown_connections 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 -- 172.21.15.79:0/4190274264 wait complete. 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 10 monclient: build_initial_monmap 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 10 monclient: monmap: 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:40:05.067 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 Processor -- start 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 1 -- start start 2024-04-23T16:40:05.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.061+0000 7fa72184d640 10 monclient: init 2024-04-23T16:40:05.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 0x7fa71c1cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6fc003770 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa71c1342c0 0x7fa71c134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): picked mon.noname-a con 0x7fa71c1342c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71c11cce0 0x7fa71c11d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): picked mon.noname-b con 0x7fa71c11cce0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c140230 con 0x7fa71c1342c0 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1400b0 con 0x7fa71c11cce0 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fa71c1403b0 con 0x7fa6fc003770 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71a7fc640 1 -- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa71c1342c0 msgr2=0x7fa71c134740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa72184d640 10 monclient(hunting): authenticate will time out at 959.256689s 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa71c1342c0 0x7fa71c134740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 0x7fa71c1cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 0x7fa71c1cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:50002/0 (socket says 172.21.15.79:50002) 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 -- 172.21.15.79:0/623700435 learned_addr learned my addr 172.21.15.79:0/623700435 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): get_auth_request con 0x7fa6fc003770 auth_method 0 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71c11cce0 0x7fa71c11d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request con 0x7fa71c11cce0 auth_method 0 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 -- 172.21.15.79:0/623700435 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71c11cce0 msgr2=0x7fa71c11d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71c11cce0 0x7fa71c11d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 -- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa71c1342c0 msgr2=0x7fa71c134740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:40:05.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa71c1342c0 0x7fa71c134740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 monclient: found mon.noname-c 2024-04-23T16:40:05.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 -- 172.21.15.79:0/623700435 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c072c70 con 0x7fa6fc003770 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 -- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 msgr2=0x7fa71c1cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.065+0000 7fa71affd640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 0x7fa71c1cf6b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa71401adb0 tx=0x7fa714019f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7140bd030 con 0x7fa6fc003770 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: dump: 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.076 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa714009430 con 0x7fa6fc003770 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7140b3040 con 0x7fa6fc003770 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 10 monclient: dump: 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.077 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 5 monclient: authenticate success, global_id 4343 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa7197fa640 4 set_mon_vals no callback set 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 monclient: _renew_subs 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa71c13cc30 con 0x7fa6fc003770 2024-04-23T16:40:05.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 monclient: _renew_subs 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71c120c70 con 0x7fa6fc003770 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fa7140c0070 con 0x7fa6fc003770 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa71c0692e0 con 0x7fa6fc003770 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 1 --2- 172.21.15.79:0/623700435 >> v2:172.21.15.204:6812/722020384 conn(0x7fa6fc0371d0 0x7fa6fc039690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (secure 0 0 0) 0x7fa7140e4670 con 0x7fa6fc003770 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa71a7fc640 1 --2- 172.21.15.79:0/623700435 >> v2:172.21.15.204:6812/722020384 conn(0x7fa6fc0371d0 0x7fa6fc039690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.069+0000 7fa71a7fc640 10 monclient: get_auth_request con 0x7fa6fc0371d0 auth_method 0 2024-04-23T16:40:05.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.073+0000 7fa71a7fc640 1 --2- 172.21.15.79:0/623700435 >> v2:172.21.15.204:6812/722020384 conn(0x7fa6fc0371d0 0x7fa6fc039690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa71c11e580 tx=0x7fa704007a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.077+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa714017140 con 0x7fa6fc003770 2024-04-23T16:40:05.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.077+0000 7fa713fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.077+0000 7fa713fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: starting. 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: creating pool ceph_test_rados_delete_pools_parallel.smithi204-26670 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: created object 0... 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: created object 25... 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: created object 49... 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: finishing. 2024-04-23T16:40:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:process_1_[26671]: shutting down. 2024-04-23T16:40:05.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa72184d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:40:05.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa72184d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:40:05.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa72184d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa71c0692e0 con 0x7fa6fc003770 2024-04-23T16:40:05.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa713fff640 1 -- 172.21.15.79:0/623700435 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fa7140e33e0 con 0x7fa6fc003770 2024-04-23T16:40:05.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa713fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:40:05.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.301+0000 7fa713fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:40:05.306 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:40:05.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 >> v2:172.21.15.204:6812/722020384 conn(0x7fa6fc0371d0 msgr2=0x7fa6fc039690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 --2- 172.21.15.79:0/623700435 >> v2:172.21.15.204:6812/722020384 conn(0x7fa6fc0371d0 0x7fa6fc039690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa71c11e580 tx=0x7fa704007a40 comp rx=0 tx=0).stop 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 10 monclient: shutdown 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 msgr2=0x7fa71c1cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 0x7fa71c1cf6b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa71401adb0 tx=0x7fa714019f20 comp rx=0 tx=0).stop 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 shutdown_connections 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 --2- 172.21.15.79:0/623700435 >> v2:172.21.15.204:6812/722020384 conn(0x7fa6fc0371d0 0x7fa6fc039690 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71c11cce0 0x7fa71c11d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa71c1342c0 0x7fa71c134740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 --2- 172.21.15.79:0/623700435 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fa6fc003770 0x7fa71c1cf6b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 >> 172.21.15.79:0/623700435 conn(0x7fa71c0a25b0 msgr2=0x7fa71c09d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 shutdown_connections 2024-04-23T16:40:05.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.305+0000 7fa72184d640 1 -- 172.21.15.79:0/623700435 wait complete. 2024-04-23T16:40:05.325 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-23T16:40:05.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-23T16:40:05.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient: get_monmap_and_config 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient: build_initial_monmap 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient: monmap: 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:40:05.450256+0000 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:40:05.450256+0000 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:40:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 Processor -- start 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 -- start start 2024-04-23T16:40:05.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient: init 2024-04-23T16:40:05.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:05.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-23T16:40:05.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 0x7fc5540a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): picked mon.noname-b con 0x7fc5540a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc554137c90 0x7fc5540a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): picked mon.noname-c con 0x7fc554137c90 addr v2:172.21.15.204:3300/0 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc554138090 addr v1:172.21.15.204:6789/0 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5540a5d40 con 0x7fc554138090 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc5540a0f10 con 0x7fc554137c90 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc5540a3f20 con 0x7fc5540a3600 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55c509640 10 monclient(hunting): authenticate will time out at 959.638900s 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 0x7fc5540a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 0x7fc5540a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:50010/0 (socket says 172.21.15.79:50010) 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 -- 172.21.15.79:0/4000864241 learned_addr learned my addr 172.21.15.79:0/4000864241 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): get_auth_request con 0x7fc5540a3600 auth_method 0 2024-04-23T16:40:05.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc559aa4640 1 --2- 172.21.15.79:0/4000864241 >> v2:172.21.15.204:3300/0 conn(0x7fc554137c90 0x7fc5540a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc559aa4640 10 monclient(hunting): get_auth_request con 0x7fc554137c90 auth_method 0 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc559aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc559aa4640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc559aa4640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 -- 172.21.15.79:0/4000864241 >> v2:172.21.15.204:3300/0 conn(0x7fc554137c90 msgr2=0x7fc5540a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/4000864241 >> v2:172.21.15.204:3300/0 conn(0x7fc554137c90 0x7fc5540a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 -- 172.21.15.79:0/4000864241 >> v1:172.21.15.204:6789/0 conn(0x7fc554138090 legacy=0x7fc5540a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:40:05.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 monclient: found mon.noname-b 2024-04-23T16:40:05.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.445+0000 7fc55a2a5640 1 -- 172.21.15.79:0/4000864241 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5540a1600 con 0x7fc5540a3600 2024-04-23T16:40:05.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc55a2a5640 1 -- 172.21.15.79:0/4000864241 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 msgr2=0x7fc5540a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/4000864241 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 0x7fc5540a39e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc55406d050 tx=0x7fc55001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75769591a2d2714a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 1 -- 172.21.15.79:0/4000864241 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500b0070 con 0x7fc5540a3600 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: dump: 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 1 -- 172.21.15.79:0/4000864241 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5500ab600 con 0x7fc5540a3600 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc55c509640 5 monclient: authenticate success, global_id 4346 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc55c509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: handle_config config(0 keys) 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc55c509640 10 monclient: get_monmap_and_config success 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 1 -- 172.21.15.79:0/4000864241 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500ab9d0 con 0x7fc5540a3600 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc55c509640 4 set_mon_vals no callback set 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.449+0000 7fc5592a3640 10 monclient: dump: 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.458 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient: shutdown 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- 172.21.15.79:0/4000864241 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 msgr2=0x7fc5540a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 --2- 172.21.15.79:0/4000864241 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 0x7fc5540a39e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc55406d050 tx=0x7fc55001c3a0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- 172.21.15.79:0/4000864241 shutdown_connections 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 --2- 172.21.15.79:0/4000864241 >> v2:172.21.15.204:3300/0 conn(0x7fc554137c90 0x7fc5540a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 --2- 172.21.15.79:0/4000864241 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5540a3600 0x7fc5540a39e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- 172.21.15.79:0/4000864241 >> 172.21.15.79:0/4000864241 conn(0x7fc5540aa3e0 msgr2=0x7fc5540a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:40:05.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- 172.21.15.79:0/4000864241 shutdown_connections 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- 172.21.15.79:0/4000864241 wait complete. 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient: build_initial_monmap 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient: monmap: 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 Processor -- start 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- start start 2024-04-23T16:40:05.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient: init 2024-04-23T16:40:05.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5540a3600 0x7fc5541cf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): picked mon.noname-b con 0x7fc5540a3600 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc554137c90 0x7fc5541cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): picked mon.noname-a con 0x7fc554137c90 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 0x7fc55413c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): picked mon.noname-c con 0x7fc554138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541422f0 con 0x7fc554137c90 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc554142170 con 0x7fc5540a3600 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc554142470 con 0x7fc554138090 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55c509640 10 monclient(hunting): authenticate will time out at 959.645967s 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc559aa4640 1 -- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc554137c90 msgr2=0x7fc5541cfe50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc559aa4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc554137c90 0x7fc5541cfe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 0x7fc55413c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 0x7fc55413c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:50020/0 (socket says 172.21.15.79:50020) 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 learned_addr learned my addr 172.21.15.79:0/3939665034 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): get_auth_request con 0x7fc554138090 auth_method 0 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5540a3600 0x7fc5541cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55a2a5640 10 monclient(hunting): get_auth_request con 0x7fc5540a3600 auth_method 0 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55a2a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55a2a5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55a2a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5540a3600 msgr2=0x7fc5541cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5540a3600 0x7fc5541cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc554137c90 msgr2=0x7fc5541cfe50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc554137c90 0x7fc5541cfe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 monclient: found mon.noname-c 2024-04-23T16:40:05.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.453+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5540a40e0 con 0x7fc554138090 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 msgr2=0x7fc55413c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55aaa6640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 0x7fc55413c710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc54c00c7b0 tx=0x7fc54c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 DispatchQueue::entry inject delay of 0.002000 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c00ac60 con 0x7fc554138090 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: dump: 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.469 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc54c00ae00 con 0x7fc554138090 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 5 monclient: authenticate success, global_id 4349 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 10 monclient: _renew_subs 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55413e310 con 0x7fc554138090 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:40:05.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c004bb0 con 0x7fc554138090 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 10 monclient: dump: 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54bfff640 4 set_mon_vals no callback set 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 10 monclient: _renew_subs 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc554136f10 con 0x7fc554138090 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 0x7fc55413cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc5540a75a0 con 0x7fc5541d0390 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 0x7fc55413cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55a2a5640 10 monclient: get_auth_request con 0x7fc5541d0390 auth_method 0 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55a2a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55a2a5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc55a2a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.457+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fc54c005030 con 0x7fc554138090 2024-04-23T16:40:05.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 --2- 172.21.15.79:0/3939665034 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5380363e0 0x7fc5380388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc554068ae0 con 0x7fc5541d0390 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (secure 0 0 0) 0x7fc54c049410 con 0x7fc554138090 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc559aa4640 1 --2- 172.21.15.79:0/3939665034 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5380363e0 0x7fc5380388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc559aa4640 10 monclient: get_auth_request con 0x7fc5380363e0 auth_method 0 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 msgr2=0x7fc55413cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 0x7fc55413cc50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5500ab000 tx=0x7fc550019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500b9020 con 0x7fc5541d0390 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 10 monclient: dump: 2024-04-23T16:40:05.473 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.474 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc5500b5480 con 0x7fc5541d0390 2024-04-23T16:40:05.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 msgr2=0x7fc55413cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 0x7fc55413cc50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5500ab000 tx=0x7fc550019140 comp rx=0 tx=0).stop 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 msgr2=0x7fc55413cc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc54affd640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc5541d0390 0x7fc55413cc50 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5500ab000 tx=0x7fc550019140 comp rx=0 tx=0).stop 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.461+0000 7fc559aa4640 1 --2- 172.21.15.79:0/3939665034 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5380363e0 0x7fc5380388a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc544006e90 tx=0x7fc544006e20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55c509640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55c509640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 0x7fc554138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55c509640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fc55413d570 con 0x7fc55413d190 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 0x7fc554138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient: get_auth_request con 0x7fc55413d190 auth_method 0 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 1 -- 172.21.15.79:0/3939665034 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc554121030 con 0x7fc55413d190 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 msgr2=0x7fc554138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc55a2a5640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 0x7fc554138570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc55001cc40 tx=0x7fc5500a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500b9030 con 0x7fc55413d190 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc54affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.469+0000 7fc54affd640 10 monclient: dump: 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.478 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 <== mon.2 v2:172.21.15.79:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fc5500c0220 con 0x7fc55413d190 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 msgr2=0x7fc554138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 0x7fc554138570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc55001cc40 tx=0x7fc5500a8000 comp rx=0 tx=0).stop 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 msgr2=0x7fc554138570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc54affd640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc55413d190 0x7fc554138570 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc55001cc40 tx=0x7fc5500a8000 comp rx=0 tx=0).stop 2024-04-23T16:40:05.479 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "peon", 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 10, 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 40307, 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 0, 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:40:05.480 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:40:05.481 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:40:05.482 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:40:05.483 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:40:05.484 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:40:05.485 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.488 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.489 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 4 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.490 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 3 2024-04-23T16:40:05.491 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.491 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.491 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:40:05.491 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:40:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5380363e0 msgr2=0x7fc5380388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 --2- 172.21.15.79:0/3939665034 >> v2:172.21.15.204:6812/722020384 conn(0x7fc5380363e0 0x7fc5380388a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc544006e90 tx=0x7fc544006e20 comp rx=0 tx=0).stop 2024-04-23T16:40:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 10 monclient: shutdown 2024-04-23T16:40:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:40:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 msgr2=0x7fc55413c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 --2- 172.21.15.79:0/3939665034 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fc554138090 0x7fc55413c710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc54c00c7b0 tx=0x7fc54c00cc70 comp rx=0 tx=0).stop 2024-04-23T16:40:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 reap_dead start 2024-04-23T16:40:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55aaa6640 1 -- 172.21.15.79:0/3939665034 reap_dead start 2024-04-23T16:40:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 shutdown_connections 2024-04-23T16:40:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 >> 172.21.15.79:0/3939665034 conn(0x7fc5540aa3e0 msgr2=0x7fc5540a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:40:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 shutdown_connections 2024-04-23T16:40:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.473+0000 7fc55c509640 1 -- 172.21.15.79:0/3939665034 wait complete. 2024-04-23T16:40:05.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-23T16:40:05.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-23T16:40:05.493 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient: get_monmap_and_config 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient: build_initial_monmap 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient: monmap: 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:40:05.609956+0000 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:40:05.609956+0000 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:40:05.608 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:40:05.609 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:40:05.609 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 1 Processor -- start 2024-04-23T16:40:05.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 1 -- start start 2024-04-23T16:40:05.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient: init 2024-04-23T16:40:05.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:05.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-04-23T16:40:05.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f180c135880 addr v1:172.21.15.204:6789/0 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f180c09fdd0 0x7f180c13a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f180c09fdd0 addr v2:172.21.15.79:3300/0 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f180c135430 addr v1:172.21.15.79:6789/0 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f180c09a440 con 0x7f180c135430 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f180c09a720 con 0x7f180c135880 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f180c0a0530 con 0x7f180c09fdd0 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f1811a2b640 10 monclient(hunting): authenticate will time out at 959.798510s 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f180b7fe640 1 -- >> v1:172.21.15.79:6789/0 conn(0x7f180c135430 legacy=0x7f180c13cca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.79:6789/0 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f180a7fc640 1 -- >> v2:172.21.15.79:3300/0 conn(0x7f180c09fdd0 msgr2=0x7f180c13a750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f180a7fc640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f180c09fdd0 0x7f180c13a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f180affd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f180c135880 0x7f180c135c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.79:42084/0 (socket says 172.21.15.79:42084) 2024-04-23T16:40:05.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.605+0000 7f180affd640 1 -- 172.21.15.79:0/1649634440 learned_addr learned my addr 172.21.15.79:0/1649634440 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:40:05.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180affd640 1 -- 172.21.15.79:0/1649634440 >> v1:172.21.15.204:6789/0 conn(0x7f180c135880 legacy=0x7f180c135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221785621 0 0) 0x7f180c09a720 con 0x7f180c135880 2024-04-23T16:40:05.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): _negotiate 2024-04-23T16:40:05.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): my global_id is 4363 2024-04-23T16:40:05.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f4003530 con 0x7f180c135880 2024-04-23T16:40:05.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3318188108 0 0) 0x7f17f4003530 con 0x7f180c135880 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 >> v2:172.21.15.79:3300/0 conn(0x7f180c09fdd0 msgr2=0x7f180c13a750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 --2- 172.21.15.79:0/1649634440 >> v2:172.21.15.79:3300/0 conn(0x7f180c09fdd0 0x7f180c13a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 >> v1:172.21.15.79:6789/0 conn(0x7f180c135430 legacy=0x7f180c13cca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 monclient: found mon.noname-c-legacy 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.204:6789/0 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c067bf0 con 0x7f180c135880 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1811a2b640 5 monclient: authenticate success, global_id 4363 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1811a2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:40:05.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1099920027 0 0) 0x7f17fc0019f0 con 0x7f180c135880 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: dump: 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.617 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] but i'm connected to v1:172.21.15.204:6789/0, reconnecting 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 >> v1:172.21.15.204:6789/0 conn(0x7f180c135880 legacy=0x7f180c135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f40039c0 0x7f17f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): picked mon.b con 0x7f17f40039c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17f4004810 0x7f17f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): picked mon.a con 0x7f17f4004810 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f17f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): picked mon.c con 0x7f17f40053d0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f180c0a0530 con 0x7f17f4004810 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17f400c270 con 0x7f17f40039c0 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f17f400c470 con 0x7f17f40053d0 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1809ffb640 10 monclient(hunting): _finish_auth 0 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f1811a2b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180a7fc640 1 -- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17f4004810 msgr2=0x7f17f4004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180a7fc640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17f4004810 0x7f17f4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f17f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): get_auth_request con 0x7f17f40053d0 auth_method 0 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180affd640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f40039c0 0x7f17f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180affd640 10 monclient(hunting): get_auth_request con 0x7f17f40039c0 auth_method 0 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient(hunting): handle_auth_done global_id 4363 payload 293 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 -- 172.21.15.79:0/1649634440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f40039c0 msgr2=0x7f17f4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f40039c0 0x7f17f4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 -- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17f4004810 msgr2=0x7f17f4004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17f4004810 0x7f17f4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 monclient: found mon.c 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 1 -- 172.21.15.79:0/1649634440 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f4003530 con 0x7f17f40053d0 2024-04-23T16:40:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.609+0000 7f180b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f180b7fe640 1 -- 172.21.15.79:0/1649634440 reap_dead start 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f180b7fe640 1 -- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 msgr2=0x7f17f400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f180b7fe640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f17f400b9d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1804009e30 tx=0x7f180401caf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d253c4bea9e7c2cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804009a30 con 0x7f17f40053d0 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: dump: 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.627 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18040a9070 con 0x7f17f40053d0 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 10 monclient: get_monmap_and_config success 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 4 set_mon_vals no callback set 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 1 -- 172.21.15.79:0/1649634440 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18040a5a50 con 0x7f17f40053d0 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1809ffb640 10 monclient: dump: 2024-04-23T16:40:05.628 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 10 monclient: shutdown 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 -- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 msgr2=0x7f17f400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f17f400b9d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1804009e30 tx=0x7f180401caf0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 -- 172.21.15.79:0/1649634440 shutdown_connections 2024-04-23T16:40:05.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 --2- 172.21.15.79:0/1649634440 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f17f400b9d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 -- 172.21.15.79:0/1649634440 >> 172.21.15.79:0/1649634440 conn(0x7f180c0a2bf0 msgr2=0x7f180c0a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 -- 172.21.15.79:0/1649634440 shutdown_connections 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.613+0000 7f1811a2b640 1 -- 172.21.15.79:0/1649634440 wait complete. 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient: build_initial_monmap 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient: monmap: 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:40:05.630 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 Processor -- start 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 -- start start 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient: init 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient: _reopen_session rank -1 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f180c134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f17f40053d0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f180c13da10 0x7f180c09d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f180c13da10 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c09db60 0x7f180c09dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f180c09db60 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f180c135ff0 con 0x7f180c13da10 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f180c135e70 con 0x7f180c09db60 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f180c136170 con 0x7f17f40053d0 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): _renew_subs 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f1811a2b640 10 monclient(hunting): authenticate will time out at 959.809535s 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180a7fc640 1 -- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f180c13da10 msgr2=0x7f180c09d620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:3300/0 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f180c13da10 0x7f180c09d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f180c134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f180c134ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:50048/0 (socket says 172.21.15.79:50048) 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 -- 172.21.15.79:0/2590173190 learned_addr learned my addr 172.21.15.79:0/2590173190 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): get_auth_request con 0x7f17f40053d0 auth_method 0 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180b7fe640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c09db60 0x7f180c09dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180b7fe640 10 monclient(hunting): get_auth_request con 0x7f180c09db60 auth_method 0 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c09db60 msgr2=0x7f180c09dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c09db60 0x7f180c09dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f180c13da10 msgr2=0x7f180c09d620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f180c13da10 0x7f180c09d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient: _finish_hunting 0 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 monclient: found mon.noname-c 2024-04-23T16:40:05.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.617+0000 7f180affd640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c073420 con 0x7f17f40053d0 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180affd640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 msgr2=0x7f180c134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180affd640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f180c134ca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17fc005e90 tx=0x7f17fc007420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00e8d0 con 0x7f17f40053d0 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: dump: 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.634 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: _finish_auth 0 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: _check_auth_tickets 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 5 monclient: authenticate success, global_id 4358 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17fc018070 con 0x7f17f40053d0 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00ecd0 con 0x7f17f40053d0 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 10 monclient: dump: 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.635 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f18097fa640 4 set_mon_vals no callback set 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 10 monclient: _renew_subs 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180c13c000 con 0x7f17f40053d0 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 10 monclient: _renew_subs 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:40:05.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f180c13de30 con 0x7f17f40053d0 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 0x7f180c1368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f17fc01d460 con 0x7f17f40053d0 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f180c09c630 con 0x7f180c136500 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 1 --2- 172.21.15.79:0/2590173190 >> v2:172.21.15.204:6812/722020384 conn(0x7f17f8036390 0x7f17f8038850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3898+0+0 (secure 0 0 0) 0x7f17fc049020 con 0x7f17f40053d0 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 0x7f180c1368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient: get_auth_request con 0x7f180c136500 auth_method 0 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180b7fe640 1 --2- 172.21.15.79:0/2590173190 >> v2:172.21.15.204:6812/722020384 conn(0x7f17f8036390 0x7f17f8038850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180b7fe640 10 monclient: get_auth_request con 0x7f17f8036390 auth_method 0 2024-04-23T16:40:05.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-04-23T16:40:05.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.621+0000 7f180a7fc640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f180c09f240 con 0x7f180c136500 2024-04-23T16:40:05.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f180b7fe640 1 --2- 172.21.15.79:0/2590173190 >> v2:172.21.15.204:6812/722020384 conn(0x7f17f8036390 0x7f17f8038850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f180c06e6c0 tx=0x7f1804005e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f180a7fc640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 msgr2=0x7f180c1368e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f180a7fc640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 0x7f180c1368e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f18000038c0 tx=0x7f180000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800016070 con 0x7f180c136500 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 10 monclient: dump: 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.640 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.1 v2:172.21.15.204:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1800005270 con 0x7f180c136500 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 msgr2=0x7f180c1368e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 0x7f180c1368e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f18000038c0 tx=0x7f180000b8e0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 msgr2=0x7f180c1368e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.625+0000 7f17f3fff640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136500 0x7f180c1368e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f1811a2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f1811a2b640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 0x7f180c1d99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f1811a2b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f180c068ae0 con 0x7f180c136e20 2024-04-23T16:40:05.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 0x7f180c1d99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient: get_auth_request con 0x7f180c136e20 auth_method 0 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 1 -- 172.21.15.79:0/2590173190 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f180c06c1f0 con 0x7f180c136e20 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 msgr2=0x7f180c1d99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.633+0000 7f180a7fc640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 0x7f180c1d99e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1800009fd0 tx=0x7f18000059a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800016050 con 0x7f180c136e20 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:40:05.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 10 monclient: dump: 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:40:05.643 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 <== mon.1 v2:172.21.15.204:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f1800017040 con 0x7f180c136e20 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 msgr2=0x7f180c1d99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 0x7f180c1d99e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1800009fd0 tx=0x7f18000059a0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 msgr2=0x7f180c1d99e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f17f3fff640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f180c136e20 0x7f180c1d99e0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1800009fd0 tx=0x7f18000059a0 comp rx=0 tx=0).stop 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:40:05.644 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "leader", 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 10, 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 40443, 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 0, 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:40:05.645 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:40:05.646 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:40:05.647 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:40:05.648 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:40:05.649 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.650 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:40:05.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:40:05.652 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.653 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.654 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 2 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.655 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 5 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": [ 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:40:05.656 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:40:05.657 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 >> v2:172.21.15.204:6812/722020384 conn(0x7f17f8036390 msgr2=0x7f17f8038850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 --2- 172.21.15.79:0/2590173190 >> v2:172.21.15.204:6812/722020384 conn(0x7f17f8036390 0x7f17f8038850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f180c06e6c0 tx=0x7f1804005e10 comp rx=0 tx=0).stop 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 10 monclient: shutdown 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 msgr2=0x7f180c134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 --2- 172.21.15.79:0/2590173190 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f17f40053d0 0x7f180c134ca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17fc005e90 tx=0x7f17fc007420 comp rx=0 tx=0).stop 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f180b7fe640 1 -- 172.21.15.79:0/2590173190 reap_dead start 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f180b7fe640 1 -- 172.21.15.79:0/2590173190 reap_dead start 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 shutdown_connections 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 >> 172.21.15.79:0/2590173190 conn(0x7f180c0a2bf0 msgr2=0x7f180c09ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 shutdown_connections 2024-04-23T16:40:05.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:40:05.637+0000 7f1811a2b640 1 -- 172.21.15.79:0/2590173190 wait complete. 2024-04-23T16:40:05.659 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-23T16:40:05.659 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-23T16:40:05.887 INFO:teuthology.orchestra.run.smithi204.stdout:process_2_[26672]: starting. 2024-04-23T16:40:05.888 INFO:teuthology.orchestra.run.smithi204.stdout:process_2_[26672]: deleting pool ceph_test_rados_delete_pools_parallel.smithi204-26670 2024-04-23T16:40:05.888 INFO:teuthology.orchestra.run.smithi204.stdout:process_2_[26672]: shutting down. 2024-04-23T16:40:08.124 INFO:teuthology.orchestra.run.smithi204.stdout:******************************* 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: starting. 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: creating pool ceph_test_rados_delete_pools_parallel.smithi204-26670 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: created object 0... 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: created object 25... 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: created object 49... 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: finishing. 2024-04-23T16:40:08.125 INFO:teuthology.orchestra.run.smithi204.stdout:process_3_[26707]: shutting down. 2024-04-23T16:40:08.184 INFO:teuthology.orchestra.run.smithi204.stdout:******************************* 2024-04-23T16:40:08.184 INFO:teuthology.orchestra.run.smithi204.stdout:process_5_[26709]: starting. 2024-04-23T16:40:08.184 INFO:teuthology.orchestra.run.smithi204.stdout:process_5_[26709]: listing objects. 2024-04-23T16:40:08.184 INFO:teuthology.orchestra.run.smithi204.stdout:process_5_[26709]: listed object 0... 2024-04-23T16:40:08.185 INFO:teuthology.orchestra.run.smithi204.stdout:process_5_[26709]: listed object 25... 2024-04-23T16:40:08.185 INFO:teuthology.orchestra.run.smithi204.stdout:process_5_[26709]: saw 50 objects 2024-04-23T16:40:08.185 INFO:teuthology.orchestra.run.smithi204.stdout:process_5_[26709]: shutting down. 2024-04-23T16:40:08.912 INFO:teuthology.orchestra.run.smithi204.stdout:******************************* 2024-04-23T16:40:08.913 INFO:teuthology.orchestra.run.smithi204.stdout:process_4_[26708]: starting. 2024-04-23T16:40:08.913 INFO:teuthology.orchestra.run.smithi204.stdout:process_4_[26708]: deleting pool ceph_test_rados_delete_pools_parallel.smithi204-26670 2024-04-23T16:40:08.913 INFO:teuthology.orchestra.run.smithi204.stdout:process_4_[26708]: shutting down. 2024-04-23T16:40:08.915 INFO:teuthology.orchestra.run.smithi204.stdout:******************************* 2024-04-23T16:40:08.915 INFO:teuthology.orchestra.run.smithi204.stdout:******************************* 2024-04-23T16:40:08.915 INFO:teuthology.orchestra.run.smithi204.stdout:******* SUCCESS ********** 2024-04-23T16:40:08.918 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-23T16:40:08.930 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-23T16:41:35.660 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-23T16:41:35.660 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-23T16:41:35.661 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-23T16:41:35.661 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-23T16:41:35.672 INFO:tasks.ceph.mon.a:Started 2024-04-23T16:41:35.672 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-23T16:41:35.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:41:35.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.786+0000 7f80954b1640 10 monclient: get_monmap_and_config 2024-04-23T16:41:35.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.786+0000 7f80954b1640 10 monclient: build_initial_monmap 2024-04-23T16:41:35.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.786+0000 7f80954b1640 10 monclient: monmap: 2024-04-23T16:41:35.792 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:35.792 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:35.792 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:35.793492+0000 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:35.793492+0000 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:35.793 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:35.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 Processor -- start 2024-04-23T16:41:35.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 -- start start 2024-04-23T16:41:35.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient: init 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 0x7f80900a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): picked mon.noname-b con 0x7f809006ca30 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f80900a3370 0x7f80900a3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): picked mon.noname-c con 0x7f80900a3370 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80900a47e0 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80900a2f20 con 0x7f80900a47e0 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f80900a1b50 con 0x7f80900a3370 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f809006ce10 con 0x7f809006ca30 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:35.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f80954b1640 10 monclient(hunting): authenticate will time out at 1049.981224s 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 0x7f80900a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 0x7f80900a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:41624/0 (socket says 172.21.15.79:41624) 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 -- 172.21.15.79:0/2463865651 learned_addr learned my addr 172.21.15.79:0/2463865651 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): get_auth_request con 0x7f809006ca30 auth_method 0 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808e7fc640 1 --2- 172.21.15.79:0/2463865651 >> v2:172.21.15.204:3300/0 conn(0x7f80900a3370 0x7f80900a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808e7fc640 10 monclient(hunting): get_auth_request con 0x7f80900a3370 auth_method 0 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:35.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 -- 172.21.15.79:0/2463865651 >> v2:172.21.15.204:3300/0 conn(0x7f80900a3370 msgr2=0x7f80900a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 --2- 172.21.15.79:0/2463865651 >> v2:172.21.15.204:3300/0 conn(0x7f80900a3370 0x7f80900a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 -- 172.21.15.79:0/2463865651 >> v1:172.21.15.204:6789/0 conn(0x7f80900a47e0 legacy=0x7f80900a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient: _finish_hunting 0 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 monclient: found mon.noname-b 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:35.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.790+0000 7f808effd640 1 -- 172.21.15.79:0/2463865651 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80900a5860 con 0x7f809006ca30 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808effd640 1 -- 172.21.15.79:0/2463865651 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 msgr2=0x7f80900a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808effd640 1 --2- 172.21.15.79:0/2463865651 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 0x7f80900a2830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8080009870 tx=0x7f808001c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b453dfe50c3290 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 1 -- 172.21.15.79:0/2463865651 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800b0070 con 0x7f809006ca30 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: dump: 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:35.800 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: _finish_auth 0 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: _check_auth_tickets 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 1 -- 172.21.15.79:0/2463865651 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80800ab600 con 0x7f809006ca30 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 1 -- 172.21.15.79:0/2463865651 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800ab9d0 con 0x7f809006ca30 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 5 monclient: authenticate success, global_id 4370 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 4 set_mon_vals no callback set 2024-04-23T16:41:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f808dffb640 10 monclient: dump: 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 10 monclient: shutdown 2024-04-23T16:41:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 -- 172.21.15.79:0/2463865651 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 msgr2=0x7f80900a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 --2- 172.21.15.79:0/2463865651 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 0x7f80900a2830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8080009870 tx=0x7f808001c500 comp rx=0 tx=0).stop 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 -- 172.21.15.79:0/2463865651 shutdown_connections 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 --2- 172.21.15.79:0/2463865651 >> v2:172.21.15.204:3300/0 conn(0x7f80900a3370 0x7f80900a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 --2- 172.21.15.79:0/2463865651 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f809006ca30 0x7f80900a2830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 -- 172.21.15.79:0/2463865651 >> 172.21.15.79:0/2463865651 conn(0x7f80900aa3e0 msgr2=0x7f80900a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 -- 172.21.15.79:0/2463865651 shutdown_connections 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 1 -- 172.21.15.79:0/2463865651 wait complete. 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 10 monclient: build_initial_monmap 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.794+0000 7f80954b1640 10 monclient: monmap: 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:35.803 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:35.804 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:35.804 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:35.804 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:35.804 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:35.804 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 Processor -- start 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 -- start start 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: init 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f809006ca30 0x7f80901cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:35.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): picked mon.noname-b con 0x7f809006ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): picked mon.noname-a con 0x7f80900a3370 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 0x7f8090122200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): picked mon.noname-c con 0x7f80900a47e0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8090142490 con 0x7f80900a3370 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8090142310 con 0x7f809006ca30 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f8090142610 con 0x7f80900a47e0 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient(hunting): authenticate will time out at 1049.988543s 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:35.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 0x7f8090122200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59948/0 (socket says 172.21.15.79:59948) 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 -- 172.21.15.79:0/1578508845 learned_addr learned my addr 172.21.15.79:0/1578508845 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 10 monclient(hunting): get_auth_request con 0x7f80900a3370 auth_method 0 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 0x7f8090122200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:41634/0 (socket says 172.21.15.79:41634) 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): get_auth_request con 0x7f80900a47e0 auth_method 0 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:35.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 -- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 msgr2=0x7f809013b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 -- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 msgr2=0x7f809013b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808e7fc640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808effd640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f809006ca30 0x7f80901cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808effd640 10 monclient(hunting): get_auth_request con 0x7f809006ca30 auth_method 0 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808effd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 -- 172.21.15.79:0/1578508845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f809006ca30 msgr2=0x7f80901cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f809006ca30 0x7f80901cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 -- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 msgr2=0x7f809013b6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T16:41:35.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 monclient: found mon.noname-c 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 -- 172.21.15.79:0/1578508845 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8090126070 con 0x7f80900a47e0 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 -- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 msgr2=0x7f8090122200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808f7fe640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 0x7f8090122200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f808800c780 tx=0x7f808800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088016020 con 0x7f80900a47e0 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: dump: 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.812 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808800ac60 con 0x7f80900a47e0 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 5 monclient: authenticate success, global_id 4373 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: _renew_subs 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80900a4000 con 0x7f80900a47e0 2024-04-23T16:41:35.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088012660 con 0x7f80900a47e0 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80777fe640 10 monclient: dump: 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:35.814 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: _renew_subs 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f808d7fa640 4 set_mon_vals no callback set 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.798+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80901d06f0 con 0x7f80900a47e0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80954b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80954b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80954b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8090068ae0 con 0x7f80900a47e0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f8088019020 con 0x7f80900a47e0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80777fe640 1 --2- 172.21.15.79:0/1578508845 >> v2:172.21.15.204:6812/722020384 conn(0x7f807c0363e0 0x7f807c0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f8088048ed0 con 0x7f80900a47e0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f808effd640 1 --2- 172.21.15.79:0/1578508845 >> v2:172.21.15.204:6812/722020384 conn(0x7f807c0363e0 0x7f807c0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:35.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.802+0000 7f808effd640 10 monclient: get_auth_request con 0x7f807c0363e0 auth_method 0 2024-04-23T16:41:35.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.806+0000 7f808effd640 1 --2- 172.21.15.79:0/1578508845 >> v2:172.21.15.204:6812/722020384 conn(0x7f807c0363e0 0x7f807c0388a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f809006d150 tx=0x7f8080008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:35.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.806+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8088020030 con 0x7f80900a47e0 2024-04-23T16:41:35.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.806+0000 7f80777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:35.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:35.806+0000 7f80777fe640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:36.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.014+0000 7f80954b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:41:36.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.014+0000 7f80954b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:41:36.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.014+0000 7f80954b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:36.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.014+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8090125850 con 0x7f80900a47e0 2024-04-23T16:41:36.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.014+0000 7f80777fe640 1 -- 172.21.15.79:0/1578508845 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f80880212b0 con 0x7f80900a47e0 2024-04-23T16:41:36.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:41:36.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80777fe640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:36.021 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 >> v2:172.21.15.204:6812/722020384 conn(0x7f807c0363e0 msgr2=0x7f807c0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 --2- 172.21.15.79:0/1578508845 >> v2:172.21.15.204:6812/722020384 conn(0x7f807c0363e0 0x7f807c0388a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f809006d150 tx=0x7f8080008040 comp rx=0 tx=0).stop 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 10 monclient: shutdown 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 msgr2=0x7f8090122200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 0x7f8090122200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f808800c780 tx=0x7f808800cc40 comp rx=0 tx=0).stop 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 shutdown_connections 2024-04-23T16:41:36.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 --2- 172.21.15.79:0/1578508845 >> v2:172.21.15.204:6812/722020384 conn(0x7f807c0363e0 0x7f807c0388a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:36.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f80900a47e0 0x7f8090122200 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:36.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80900a3370 0x7f809013b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:36.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 --2- 172.21.15.79:0/1578508845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f809006ca30 0x7f80901cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:36.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 >> 172.21.15.79:0/1578508845 conn(0x7f80900aa3e0 msgr2=0x7f80900a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:36.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 shutdown_connections 2024-04-23T16:41:36.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:36.018+0000 7f80954b1640 1 -- 172.21.15.79:0/1578508845 wait complete. 2024-04-23T16:41:39.039 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:41:39.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient: get_monmap_and_config 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient: monmap: 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:39.170717+0000 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:39.170717+0000 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:39.170 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:39.171 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:39.171 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:39.171 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 Processor -- start 2024-04-23T16:41:39.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 -- start start 2024-04-23T16:41:39.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient: init 2024-04-23T16:41:39.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0448135a20 0x7f0448135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): picked mon.noname-a con 0x7f0448135a20 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f044809f430 0x7f044809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): picked mon.noname-c con 0x7f044809f430 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 0x7f044809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): picked mon.noname-b con 0x7f0448135300 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f0448072f80 con 0x7f0448135a20 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f044809fb10 con 0x7f044809f430 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f044809fdf0 con 0x7f0448135300 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f044e13d640 10 monclient(hunting): authenticate will time out at 1053.358616s 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 0x7f044809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f0448135a20 0x7f0448135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 0x7f044809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45356/0 (socket says 172.21.15.79:45356) 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 -- 172.21.15.79:0/2281673056 learned_addr learned my addr 172.21.15.79:0/2281673056 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): get_auth_request con 0x7f0448135300 auth_method 0 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): get_auth_request con 0x7f0448135a20 auth_method 0 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0446ffd640 1 --2- 172.21.15.79:0/2281673056 >> v2:172.21.15.204:3300/0 conn(0x7f044809f430 0x7f044809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0446ffd640 10 monclient(hunting): get_auth_request con 0x7f044809f430 auth_method 0 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0446ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0446ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 -- 172.21.15.79:0/2281673056 >> v2:172.21.15.204:3300/0 conn(0x7f044809f430 msgr2=0x7f044809ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 --2- 172.21.15.79:0/2281673056 >> v2:172.21.15.204:3300/0 conn(0x7f044809f430 0x7f044809ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 -- 172.21.15.79:0/2281673056 >> v2:172.21.15.79:3300/0 conn(0x7f0448135a20 msgr2=0x7f0448135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 --2- 172.21.15.79:0/2281673056 >> v2:172.21.15.79:3300/0 conn(0x7f0448135a20 0x7f0448135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 monclient: found mon.noname-b 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f0447fff640 1 -- 172.21.15.79:0/2281673056 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044806de30 con 0x7f0448135300 2024-04-23T16:41:39.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.166+0000 7f04477fe640 1 --2- 172.21.15.79:0/2281673056 >> v2:172.21.15.79:3300/0 conn(0x7f0448135a20 0x7f0448135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f0447fff640 1 -- 172.21.15.79:0/2281673056 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 msgr2=0x7f044809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f0447fff640 1 --2- 172.21.15.79:0/2281673056 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 0x7f044809bf40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f044806dbd0 tx=0x7f044001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc2eef486a071945 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 1 -- 172.21.15.79:0/2281673056 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04400b0070 con 0x7f0448135300 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: dump: 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f044e13d640 5 monclient: authenticate success, global_id 4376 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f044e13d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 1 -- 172.21.15.79:0/2281673056 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04400ab600 con 0x7f0448135300 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f044e13d640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f044e13d640 4 set_mon_vals no callback set 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 1 -- 172.21.15.79:0/2281673056 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04400ab9d0 con 0x7f0448135300 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.170+0000 7f04467fc640 10 monclient: dump: 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.183 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient: shutdown 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- 172.21.15.79:0/2281673056 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 msgr2=0x7f044809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- 172.21.15.79:0/2281673056 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 0x7f044809bf40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f044806dbd0 tx=0x7f044001c3e0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- 172.21.15.79:0/2281673056 shutdown_connections 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- 172.21.15.79:0/2281673056 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135300 0x7f044809bf40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- 172.21.15.79:0/2281673056 >> v2:172.21.15.204:3300/0 conn(0x7f044809f430 0x7f044809ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- 172.21.15.79:0/2281673056 >> v2:172.21.15.79:3300/0 conn(0x7f0448135a20 0x7f0448135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- 172.21.15.79:0/2281673056 >> 172.21.15.79:0/2281673056 conn(0x7f04480a25b0 msgr2=0x7f04480a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- 172.21.15.79:0/2281673056 shutdown_connections 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- 172.21.15.79:0/2281673056 wait complete. 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient: monmap: 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:39.185 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 Processor -- start 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- start start 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient: init 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f044809f430 0x7f04481cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): picked mon.noname-a con 0x7f044809f430 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0448135300 0x7f0448134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): picked mon.noname-b con 0x7f0448135300 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 0x7f0448134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): picked mon.noname-c con 0x7f0448135a20 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f044813f0a0 con 0x7f044809f430 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f044809c890 con 0x7f0448135300 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f044813f220 con 0x7f0448135a20 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f044e13d640 10 monclient(hunting): authenticate will time out at 1053.365869s 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 0x7f0448134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f044809f430 0x7f04481cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 0x7f0448134c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45364/0 (socket says 172.21.15.79:45364) 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): get_auth_request con 0x7f044809f430 auth_method 0 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 -- 172.21.15.79:0/2338554212 learned_addr learned my addr 172.21.15.79:0/2338554212 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): get_auth_request con 0x7f0448135a20 auth_method 0 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0446ffd640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0448135300 0x7f0448134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-04-23T16:41:39.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 -- 172.21.15.79:0/2338554212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0448135300 msgr2=0x7f0448134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0448135300 0x7f0448134710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 -- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f044809f430 msgr2=0x7f04481cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f044809f430 0x7f04481cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 monclient: found mon.noname-c 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0447fff640 1 -- 172.21.15.79:0/2338554212 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0448071b00 con 0x7f0448135a20 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0446ffd640 10 monclient: get_auth_request con 0x7f0448135300 auth_method 0 2024-04-23T16:41:39.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f0446ffd640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0448135300 0x7f0448134710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.174+0000 7f04477fe640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f044809f430 0x7f04481cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f0447fff640 1 -- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 msgr2=0x7f0448134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f0447fff640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 0x7f0448134c50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04400ae040 tx=0x7f0440019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04400b0040 con 0x7f0448135a20 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: dump: 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.190 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04400b35b0 con 0x7f0448135a20 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04400b39c0 con 0x7f0448135a20 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 10 monclient: dump: 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.191 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 5 monclient: authenticate success, global_id 4379 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 monclient: _renew_subs 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f0445ffb640 4 set_mon_vals no callback set 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0448072c70 con 0x7f0448135a20 2024-04-23T16:41:39.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 monclient: _renew_subs 2024-04-23T16:41:39.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0448121600 con 0x7f0448135a20 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04481d0400 con 0x7f0448135a20 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f04400bb020 con 0x7f0448135a20 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 1 --2- 172.21.15.79:0/2338554212 >> v2:172.21.15.204:6812/722020384 conn(0x7f04280363e0 0x7f04280388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f04400e2520 con 0x7f0448135a20 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f04477fe640 1 --2- 172.21.15.79:0/2338554212 >> v2:172.21.15.204:6812/722020384 conn(0x7f04280363e0 0x7f04280388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.178+0000 7f04477fe640 10 monclient: get_auth_request con 0x7f04280363e0 auth_method 0 2024-04-23T16:41:39.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.182+0000 7f04477fe640 1 --2- 172.21.15.79:0/2338554212 >> v2:172.21.15.204:6812/722020384 conn(0x7f04280363e0 0x7f04280388a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0430009580 tx=0x7f0430009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.182+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f04400c2020 con 0x7f0448135a20 2024-04-23T16:41:39.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.182+0000 7f043ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.182+0000 7f043ffff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:39.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f044e13d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:41:39.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f044e13d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:41:39.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f044e13d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f04480692e0 con 0x7f0448135a20 2024-04-23T16:41:39.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f043ffff640 1 -- 172.21.15.79:0/2338554212 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f04400160f0 con 0x7f0448135a20 2024-04-23T16:41:39.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f043ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:41:39.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.410+0000 7f043ffff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:39.417 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 >> v2:172.21.15.204:6812/722020384 conn(0x7f04280363e0 msgr2=0x7f04280388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 --2- 172.21.15.79:0/2338554212 >> v2:172.21.15.204:6812/722020384 conn(0x7f04280363e0 0x7f04280388a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0430009580 tx=0x7f0430009190 comp rx=0 tx=0).stop 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 10 monclient: shutdown 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 msgr2=0x7f0448134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 0x7f0448134c50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04400ae040 tx=0x7f0440019180 comp rx=0 tx=0).stop 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 shutdown_connections 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 --2- 172.21.15.79:0/2338554212 >> v2:172.21.15.204:6812/722020384 conn(0x7f04280363e0 0x7f04280388a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f0448135a20 0x7f0448134c50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0448135300 0x7f0448134710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 --2- 172.21.15.79:0/2338554212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f044809f430 0x7f04481cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 >> 172.21.15.79:0/2338554212 conn(0x7f04480a25b0 msgr2=0x7f0448136430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 shutdown_connections 2024-04-23T16:41:39.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.414+0000 7f044e13d640 1 -- 172.21.15.79:0/2338554212 wait complete. 2024-04-23T16:41:39.435 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-23T16:41:39.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-23T16:41:39.435 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-23T16:41:39.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient: get_monmap_and_config 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient: monmap: 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:39.545788+0000 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:39.545788+0000 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:39.545 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:39.546 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 Processor -- start 2024-04-23T16:41:39.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 -- start start 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient: init 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1dcc135910 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1dcc0a0950 0x7f1dcc13a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc0a0950 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f1dcc13ad40 0x7f1dcc13d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc13ad40 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dcc136400 con 0x7f1dcc135910 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f1dcc0a1b00 con 0x7f1dcc13ad40 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1dcc0a0d90 con 0x7f1dcc0a0950 2024-04-23T16:41:39.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd40e7640 10 monclient(hunting): authenticate will time out at 1053.733722s 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f1dcc13ad40 0x7f1dcc13d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd1e83640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f1dcc135910 0x7f1dcc135cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33370/0 (socket says 172.21.15.79:33370) 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd1e83640 1 -- 172.21.15.79:0/892994222 learned_addr learned my addr 172.21.15.79:0/892994222 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): get_auth_request con 0x7f1dcc13ad40 auth_method 0 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd1682640 1 --2- 172.21.15.79:0/892994222 >> v2:172.21.15.204:3300/0 conn(0x7f1dcc0a0950 0x7f1dcc13a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient(hunting): handle_auth_done global_id 14103 payload 274 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 >> v2:172.21.15.204:3300/0 conn(0x7f1dcc0a0950 msgr2=0x7f1dcc13a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 1 --2- 172.21.15.79:0/892994222 >> v2:172.21.15.204:3300/0 conn(0x7f1dcc0a0950 0x7f1dcc13a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 >> v1:172.21.15.79:6789/0 conn(0x7f1dcc135910 legacy=0x7f1dcc135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 1 monclient: found mon.noname-a 2024-04-23T16:41:39.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:39.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.542+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc1354e0 con 0x7f1dcc13ad40 2024-04-23T16:41:39.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 msgr2=0x7f1dcc13d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd2684640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc13d130 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0a54b0 tx=0x7f1dc801c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82f74dde6881be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 msgr2=0x7f1dcc13d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-23T16:41:39.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 msgr2=0x7f1dcc13d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T16:41:39.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd2684640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc13d130 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0a54b0 tx=0x7f1dc801c7b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:41:39.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd2684640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc13d130 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0a54b0 tx=0x7f1dc801c7b0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient: ms_handle_reset current mon [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 1 -- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 msgr2=0x7f1dcc13d130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc13d130 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient: _add_conns ranks=[1,2] 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1db8003510 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1db8007440 0x7f1db8009900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient(hunting): picked mon.noname-b con 0x7f1db8007440 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 1 -- 172.21.15.79:0/892994222 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db800a1f0 con 0x7f1db8003510 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 1 -- 172.21.15.79:0/892994222 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f1db800b430 con 0x7f1db8007440 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd0e81640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1db8007440 0x7f1db8009900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): get_auth_request con 0x7f1db8007440 auth_method 0 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient(hunting): handle_auth_done global_id 14103 payload 293 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 1 -- 172.21.15.79:0/892994222 >> v1:172.21.15.204:6789/0 conn(0x7f1db8003510 legacy=0x7f1db8005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 1 monclient: found mon.noname-b 2024-04-23T16:41:39.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.546+0000 7f1dd1682640 1 -- 172.21.15.79:0/892994222 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db800b6f0 con 0x7f1db8007440 2024-04-23T16:41:39.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1682640 1 -- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1db8007440 msgr2=0x7f1db8009900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1682640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1db8007440 0x7f1db8009900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00a910 tx=0x7f1dc8009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc5a4ddd47f90ed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 1 -- 172.21.15.79:0/892994222 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc015070 con 0x7f1db8007440 2024-04-23T16:41:39.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:39.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: dump: 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 5 monclient: authenticate success, global_id 14103 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 1 -- 172.21.15.79:0/892994222 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dbc004220 con 0x7f1db8007440 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 4 set_mon_vals no callback set 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 1 -- 172.21.15.79:0/892994222 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc00a590 con 0x7f1db8007440 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd0e81640 10 monclient: dump: 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.563 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: shutdown 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1db8007440 msgr2=0x7f1db8009900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1db8007440 0x7f1db8009900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00a910 tx=0x7f1dc8009750 comp rx=0 tx=0).stop 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd2684640 1 -- 172.21.15.79:0/892994222 reap_dead start 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- 172.21.15.79:0/892994222 shutdown_connections 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/892994222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc13d130 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- 172.21.15.79:0/892994222 >> 172.21.15.79:0/892994222 conn(0x7f1dcc0a8680 msgr2=0x7f1dcc0a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- 172.21.15.79:0/892994222 shutdown_connections 2024-04-23T16:41:39.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- 172.21.15.79:0/892994222 wait complete. 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: monmap: 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.565 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.566 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:39.566 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:39.566 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:39.566 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 Processor -- start 2024-04-23T16:41:39.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- start start 2024-04-23T16:41:39.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: init 2024-04-23T16:41:39.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:41:39.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc13ad40 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc06ba20 0x7f1dcc06be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc06ba20 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 0x7f1dcc1d60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc06c3a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc13fe90 con 0x7f1dcc13ad40 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc0a15a0 con 0x7f1dcc06ba20 2024-04-23T16:41:39.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f1dcc140010 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd40e7640 10 monclient(hunting): authenticate will time out at 1053.743844s 2024-04-23T16:41:39.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd2684640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 0x7f1dcc1d60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53576/0 (socket says 172.21.15.79:53576) 2024-04-23T16:41:39.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 1 -- 172.21.15.79:0/2789540663 learned_addr learned my addr 172.21.15.79:0/2789540663 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 10 monclient(hunting): get_auth_request con 0x7f1dcc13ad40 auth_method 0 2024-04-23T16:41:39.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd1e83640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd2684640 10 monclient(hunting): get_auth_request con 0x7f1dcc06c3a0 auth_method 0 2024-04-23T16:41:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd2684640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd2684640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.550+0000 7f1dd2684640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-04-23T16:41:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc06ba20 msgr2=0x7f1dcc06be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc06ba20 0x7f1dcc06be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 msgr2=0x7f1dcc134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc134c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 monclient: found mon.noname-c 2024-04-23T16:41:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc0a6a50 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc13ad40 0x7f1dcc134c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 msgr2=0x7f1dcc1d60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 0x7f1dcc1d60a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0a5710 tx=0x7f1dc8019550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc80bc020 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd2684640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 0x7f1dcc1d60a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0a5710 tx=0x7f1dc8019550 comp rx=0 tx=0).queue_received will delay after 1.71389e+09 on 0x7f1dc8004410 config(0 keys) 2024-04-23T16:41:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: dump: 2024-04-23T16:41:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 5 monclient: authenticate success, global_id 4382 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dc8004410 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 10 monclient: _renew_subs 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dcc072de0 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc3fff640 4 set_mon_vals no callback set 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 10 monclient: _renew_subs 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dcc138dc0 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 0x7f1dcc09ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc137910 con 0x7f1dcc138f90 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc80b2070 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 0x7f1dcc09ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 10 monclient: dump: 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f1dc80bf020 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient: get_auth_request con 0x7f1dcc138f90 auth_method 0 2024-04-23T16:41:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dc27fc640 1 --2- 172.21.15.79:0/2789540663 >> v2:172.21.15.204:6812/722020384 conn(0x7f1db0036390 0x7f1db0038850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_done global_id 14109 payload 824 2024-04-23T16:41:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.554+0000 7f1dd1e83640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1dcc138bc0 con 0x7f1dcc138f90 2024-04-23T16:41:39.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dd1682640 1 --2- 172.21.15.79:0/2789540663 >> v2:172.21.15.204:6812/722020384 conn(0x7f1db0036390 0x7f1db0038850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dd1682640 10 monclient: get_auth_request con 0x7f1db0036390 auth_method 0 2024-04-23T16:41:39.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dd1e83640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 msgr2=0x7f1dcc09ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dd1e83640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 0x7f1dcc09ba00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40072a0 tx=0x7f1dc4007760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc400fd60 con 0x7f1dcc138f90 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 10 monclient: dump: 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.586 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.0 v2:172.21.15.79:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1dc400fd60 con 0x7f1dcc138f90 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 msgr2=0x7f1dcc09ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 0x7f1dcc09ba00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40072a0 tx=0x7f1dc4007760 comp rx=0 tx=0).stop 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 msgr2=0x7f1dcc09ba00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dc27fc640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc138f90 0x7f1dcc09ba00 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40072a0 tx=0x7f1dc4007760 comp rx=0 tx=0).stop 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.558+0000 7f1dd1682640 1 --2- 172.21.15.79:0/2789540663 >> v2:172.21.15.204:6812/722020384 conn(0x7f1db0036390 0x7f1db0038850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1db8007820 tx=0x7f1dbc011000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 0x7f1dcc09c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc13a570 con 0x7f1dcc09bf40 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 0x7f1dcc09c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient: get_auth_request con 0x7f1dcc09bf40 auth_method 0 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 10 monclient(hunting): handle_auth_done global_id 14112 payload 824 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 1 -- 172.21.15.79:0/2789540663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1dcc068ae0 con 0x7f1dcc09bf40 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 msgr2=0x7f1dcc09c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd1e83640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 0x7f1dcc09c350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40072a0 tx=0x7f1dc4004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4006030 con 0x7f1dcc09bf40 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 10 monclient: dump: 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.0 v2:172.21.15.79:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f1dc4006030 con 0x7f1dcc09bf40 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 msgr2=0x7f1dcc09c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 0x7f1dcc09c350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40072a0 tx=0x7f1dc4004950 comp rx=0 tx=0).stop 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 msgr2=0x7f1dcc09c350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1dcc09bf40 0x7f1dcc09c350 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "leader", 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 14, 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 3819, 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: 0, 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 2, 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.591 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.592 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:41:39.593 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.594 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:41:39.595 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:41:39.596 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.597 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:41:39.598 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:41:39.599 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.600 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:41:39.601 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:39.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 >> v2:172.21.15.204:6812/722020384 conn(0x7f1db0036390 msgr2=0x7f1db0038850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/2789540663 >> v2:172.21.15.204:6812/722020384 conn(0x7f1db0036390 0x7f1db0038850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1db8007820 tx=0x7f1dbc011000 comp rx=0 tx=0).stop 2024-04-23T16:41:39.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 10 monclient: shutdown 2024-04-23T16:41:39.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 msgr2=0x7f1dcc1d60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd2684640 1 -- 172.21.15.79:0/2789540663 reap_dead start 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 0x7f1dcc1d60a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0a5710 tx=0x7f1dc8019550 comp rx=0 tx=0).stop 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dc27fc640 1 -- 172.21.15.79:0/2789540663 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (unknown 0 0 0) 0x7f1dc80e42f0 con 0x7f1dcc06c3a0 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 shutdown_connections 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 --2- 172.21.15.79:0/2789540663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f1dcc06c3a0 0x7f1dcc1d60a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 >> 172.21.15.79:0/2789540663 conn(0x7f1dcc0a8680 msgr2=0x7f1dcc13b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 shutdown_connections 2024-04-23T16:41:39.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.566+0000 7f1dd40e7640 1 -- 172.21.15.79:0/2789540663 wait complete. 2024-04-23T16:41:39.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-23T16:41:39.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-23T16:41:39.603 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-23T16:41:39.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient: get_monmap_and_config 2024-04-23T16:41:39.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient: monmap: 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:39.705798+0000 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:39.705798+0000 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:39.705 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:39.706 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:39.706 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 Processor -- start 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 -- start start 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient: init 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8072d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): picked mon.noname-b con 0x7fbcb81356c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fbcb8135aa0 0x7fbcb8135e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): picked mon.noname-a con 0x7fbcb8135aa0 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcb80a2540 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcb8072340 con 0x7fbcb80a2540 2024-04-23T16:41:39.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fbcb8073260 con 0x7fbcb8135aa0 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb80758f0 con 0x7fbcb81356c0 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcbcd43640 10 monclient(hunting): authenticate will time out at 1053.893662s 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8072d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45382/0 (socket says 172.21.15.79:45382) 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 -- 172.21.15.79:0/3360276897 learned_addr learned my addr 172.21.15.79:0/3360276897 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): get_auth_request con 0x7fbcb81356c0 auth_method 0 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 1 --2- 172.21.15.79:0/3360276897 >> v2:172.21.15.79:3300/0 conn(0x7fbcb8135aa0 0x7fbcb8135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): get_auth_request con 0x7fbcb8135aa0 auth_method 0 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 -- 172.21.15.79:0/3360276897 >> v2:172.21.15.79:3300/0 conn(0x7fbcb8135aa0 msgr2=0x7fbcb8135e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 --2- 172.21.15.79:0/3360276897 >> v2:172.21.15.79:3300/0 conn(0x7fbcb8135aa0 0x7fbcb8135e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 -- 172.21.15.79:0/3360276897 >> v1:172.21.15.79:6789/0 conn(0x7fbcb80a2540 legacy=0x7fbcb80a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 monclient: found mon.noname-b 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb659c640 1 -- 172.21.15.79:0/3360276897 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb8073610 con 0x7fbcb81356c0 2024-04-23T16:41:39.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.702+0000 7fbcb5d9b640 1 --2- 172.21.15.79:0/3360276897 >> v2:172.21.15.79:3300/0 conn(0x7fbcb8135aa0 0x7fbcb8135e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb659c640 1 -- 172.21.15.79:0/3360276897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 msgr2=0x7fbcb8072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb659c640 1 --2- 172.21.15.79:0/3360276897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8072d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009e80 tx=0x7fbcb001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a36585b42ec676d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 1 -- 172.21.15.79:0/3360276897 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00a5620 con 0x7fbcb81356c0 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: dump: 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.721 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 5 monclient: authenticate success, global_id 4385 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 1 -- 172.21.15.79:0/3360276897 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb00a57c0 con 0x7fbcb81356c0 2024-04-23T16:41:39.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 1 -- 172.21.15.79:0/3360276897 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00a5b90 con 0x7fbcb81356c0 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 4 set_mon_vals no callback set 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcb559a640 10 monclient: dump: 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.723 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 10 monclient: shutdown 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 -- 172.21.15.79:0/3360276897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 msgr2=0x7fbcb8072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/3360276897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8072d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009e80 tx=0x7fbcb001c890 comp rx=0 tx=0).stop 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 -- 172.21.15.79:0/3360276897 shutdown_connections 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/3360276897 >> v2:172.21.15.79:3300/0 conn(0x7fbcb8135aa0 0x7fbcb8135e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/3360276897 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8072d20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 -- 172.21.15.79:0/3360276897 >> 172.21.15.79:0/3360276897 conn(0x7fbcb80a7220 msgr2=0x7fbcb80a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 -- 172.21.15.79:0/3360276897 shutdown_connections 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 1 -- 172.21.15.79:0/3360276897 wait complete. 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.706+0000 7fbcbcd43640 10 monclient: monmap: 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 Processor -- start 2024-04-23T16:41:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 -- start start 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient: init 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): picked mon.noname-a con 0x7fbcb80a2540 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): picked mon.noname-c con 0x7fbcb81356c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcb8135aa0 0x7fbcb806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): picked mon.noname-b con 0x7fbcb8135aa0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb81401f0 con 0x7fbcb80a2540 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb8140070 con 0x7fbcb8135aa0 2024-04-23T16:41:39.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb8140370 con 0x7fbcb81356c0 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcbcd43640 10 monclient(hunting): authenticate will time out at 1053.900679s 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53600/0 (socket says 172.21.15.79:53600) 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 learned_addr learned my addr 172.21.15.79:0/198617331 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): get_auth_request con 0x7fbcb80a2540 auth_method 0 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): get_auth_request con 0x7fbcb81356c0 auth_method 0 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient(hunting): handle_auth_done global_id 14118 payload 824 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 msgr2=0x7fbcb8131660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbcb8131660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcb8135aa0 msgr2=0x7fbcb806ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcb8135aa0 0x7fbcb806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 monclient: found mon.noname-a 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb80a0c70 con 0x7fbcb80a2540 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 msgr2=0x7fbcb813c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00af040 tx=0x7fbcb0019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 msgr2=0x7fbcb813c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 msgr2=0x7fbcb813c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00af040 tx=0x7fbcb0019630 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00af040 tx=0x7fbcb0019630 comp rx=0 tx=0).stop 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 msgr2=0x7fbcb813c810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcb80a2540 0x7fbcb813c810 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcb8135aa0 0x7fbc980054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbcb8135aa0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:39.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbc98007f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbcb81356c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc980089c0 0x7fbc9800ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbc980089c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9800b950 con 0x7fbc980089c0 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9800bad0 con 0x7fbcb8135aa0 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9800bcd0 con 0x7fbcb81356c0 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbca6ffd640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbc98007f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc980089c0 0x7fbc9800ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): get_auth_request con 0x7fbcb81356c0 auth_method 0 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): get_auth_request con 0x7fbc980089c0 auth_method 0 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient(hunting): handle_auth_done global_id 14118 payload 293 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcb8135aa0 msgr2=0x7fbc980054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcb8135aa0 0x7fbc980054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc980089c0 msgr2=0x7fbc9800ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc980089c0 0x7fbc9800ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 monclient: found mon.noname-c 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb5d9b640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9800bf90 con 0x7fbcb81356c0 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.710+0000 7fbcb6d9d640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc980089c0 0x7fbc9800ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb5d9b640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 msgr2=0x7fbc98007f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb5d9b640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbc98007f60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009060 tx=0x7fbcb00a7070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac009950 con 0x7fbcb81356c0 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: dump: 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcac004020 con 0x7fbcb81356c0 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0043f0 con 0x7fbcb81356c0 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 10 monclient: dump: 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.734 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 5 monclient: authenticate success, global_id 14118 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 10 monclient: _renew_subs 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb8070430 con 0x7fbcb81356c0 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb4d99640 4 set_mon_vals no callback set 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 10 monclient: _renew_subs 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb81cfb40 con 0x7fbcb81356c0 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 0x7fbcb8131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb80a0180 con 0x7fbcb813c810 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 0x7fbcb8131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient: get_auth_request con 0x7fbcb813c810 auth_method 0 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fbcac012040 con 0x7fbcb81356c0 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> v2:172.21.15.204:6812/722020384 conn(0x7fbcb80a2540 0x7fbc98040b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcb804fcf0 con 0x7fbcb813c810 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fbcac047ff0 con 0x7fbcb81356c0 2024-04-23T16:41:39.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb6d9d640 1 --2- 172.21.15.79:0/198617331 >> v2:172.21.15.204:6812/722020384 conn(0x7fbcb80a2540 0x7fbc98040b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.714+0000 7fbcb6d9d640 10 monclient: get_auth_request con 0x7fbcb80a2540 auth_method 0 2024-04-23T16:41:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 msgr2=0x7fbcb8131660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 0x7fbcb8131660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbcb001c8d0 tx=0x7fbcb00a5850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00b6860 con 0x7fbcb813c810 2024-04-23T16:41:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 10 monclient: dump: 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbcb00b6860 con 0x7fbcb813c810 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 msgr2=0x7fbcb8131660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 0x7fbcb8131660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbcb001c8d0 tx=0x7fbcb00a5850 comp rx=0 tx=0).stop 2024-04-23T16:41:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 msgr2=0x7fbcb8131660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb813c810 0x7fbcb8131660 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbcb001c8d0 tx=0x7fbcb00a5850 comp rx=0 tx=0).stop 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.718+0000 7fbcb6d9d640 1 --2- 172.21.15.79:0/198617331 >> v2:172.21.15.204:6812/722020384 conn(0x7fbcb80a2540 0x7fbc98040b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbca8007fd0 tx=0x7fbca8007bb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcbcd43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 0x7fbcb806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcbcd43640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb806cae0 con 0x7fbcb8131ba0 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 0x7fbcb806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient: get_auth_request con 0x7fbcb8131ba0 auth_method 0 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.722+0000 7fbcb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcb659c640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-04-23T16:41:39.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbcb811dd20 con 0x7fbcb8131ba0 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcb659c640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 msgr2=0x7fbcb806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcb659c640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 0x7fbcb806ecf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80742e0 tx=0x7fbcb00b9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00caa90 con 0x7fbcb8131ba0 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 10 monclient: dump: 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.742 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 <== mon.2 v2:172.21.15.79:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7fbcb00c3460 con 0x7fbcb8131ba0 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 msgr2=0x7fbcb806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 0x7fbcb806ecf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80742e0 tx=0x7fbcb00b9000 comp rx=0 tx=0).stop 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 msgr2=0x7fbcb806ecf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbca6ffd640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb8131ba0 0x7fbcb806ecf0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80742e0 tx=0x7fbcb00b9000 comp rx=0 tx=0).stop 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:41:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "peon", 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 14, 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 134564, 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: 0, 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 2, 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:41:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:41:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 3 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 3 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:41:39.754 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 >> v2:172.21.15.204:6812/722020384 conn(0x7fbcb80a2540 msgr2=0x7fbc98040b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/198617331 >> v2:172.21.15.204:6812/722020384 conn(0x7fbcb80a2540 0x7fbc98040b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbca8007fd0 tx=0x7fbca8007bb0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 10 monclient: shutdown 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 msgr2=0x7fbc98007f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 --2- 172.21.15.79:0/198617331 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbcb81356c0 0x7fbc98007f60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009060 tx=0x7fbcb00a7070 comp rx=0 tx=0).stop 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcb6d9d640 1 -- 172.21.15.79:0/198617331 reap_dead start 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcb6d9d640 1 -- 172.21.15.79:0/198617331 reap_dead start 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 shutdown_connections 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 >> 172.21.15.79:0/198617331 conn(0x7fbcb80a7220 msgr2=0x7fbcb80a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 shutdown_connections 2024-04-23T16:41:39.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.726+0000 7fbcbcd43640 1 -- 172.21.15.79:0/198617331 wait complete. 2024-04-23T16:41:39.757 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-23T16:41:39.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-23T16:41:39.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-23T16:41:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient: get_monmap_and_config 2024-04-23T16:41:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient: monmap: 2024-04-23T16:41:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:39.866485+0000 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:39.866485+0000 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:39.866 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 Processor -- start 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 -- start start 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient: init 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9f1c1355c0 0x7f9f1c1359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): picked mon.noname-c con 0x7f9f1c1355c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f1c135ee0 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 0x7f9f1c0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): picked mon.noname-b con 0x7f9f1c09c690 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f1c09bdc0 con 0x7f9f1c135ee0 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9f1c072bb0 con 0x7f9f1c1355c0 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1c09f6c0 con 0x7f9f1c09c690 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f22889640 10 monclient(hunting): authenticate will time out at 1054.054314s 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 0x7f9f1c0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 0x7f9f1c0a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45426/0 (socket says 172.21.15.79:45426) 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 -- 172.21.15.79:0/418686330 learned_addr learned my addr 172.21.15.79:0/418686330 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): get_auth_request con 0x7f9f1c09c690 auth_method 0 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 1 --2- 172.21.15.79:0/418686330 >> v2:172.21.15.204:3300/0 conn(0x7f9f1c1355c0 0x7f9f1c1359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): get_auth_request con 0x7f9f1c1355c0 auth_method 0 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient(hunting): handle_auth_done global_id 4397 payload 274 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 -- 172.21.15.79:0/418686330 >> v2:172.21.15.204:3300/0 conn(0x7f9f1c1355c0 msgr2=0x7f9f1c1359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 --2- 172.21.15.79:0/418686330 >> v2:172.21.15.204:3300/0 conn(0x7f9f1c1355c0 0x7f9f1c1359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 -- 172.21.15.79:0/418686330 >> v1:172.21.15.204:6789/0 conn(0x7f9f1c135ee0 legacy=0x7f9f1c0a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 monclient: found mon.noname-b 2024-04-23T16:41:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.862+0000 7f9f20e26640 1 -- 172.21.15.79:0/418686330 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c09f1e0 con 0x7f9f1c09c690 2024-04-23T16:41:39.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f20e26640 1 -- 172.21.15.79:0/418686330 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 msgr2=0x7f9f1c0a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f20e26640 1 --2- 172.21.15.79:0/418686330 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 0x7f9f1c0a1af0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009e50 tx=0x7f9f1401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85cebe1cf90e202e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 1 -- 172.21.15.79:0/418686330 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140a5640 con 0x7f9f1c09c690 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: dump: 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.878 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 1 -- 172.21.15.79:0/418686330 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f140a57e0 con 0x7f9f1c09c690 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 1 -- 172.21.15.79:0/418686330 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140a5bb0 con 0x7f9f1c09c690 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 5 monclient: authenticate success, global_id 4397 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f1b7fe640 10 monclient: dump: 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.879 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 4 set_mon_vals no callback set 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 10 monclient: shutdown 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 1 -- 172.21.15.79:0/418686330 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 msgr2=0x7f9f1c0a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.866+0000 7f9f22889640 1 --2- 172.21.15.79:0/418686330 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 0x7f9f1c0a1af0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009e50 tx=0x7f9f1401c8b0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- 172.21.15.79:0/418686330 shutdown_connections 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 --2- 172.21.15.79:0/418686330 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c09c690 0x7f9f1c0a1af0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 --2- 172.21.15.79:0/418686330 >> v2:172.21.15.204:3300/0 conn(0x7f9f1c1355c0 0x7f9f1c1359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- 172.21.15.79:0/418686330 >> 172.21.15.79:0/418686330 conn(0x7f9f1c0a6250 msgr2=0x7f9f1c0a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- 172.21.15.79:0/418686330 shutdown_connections 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- 172.21.15.79:0/418686330 wait complete. 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient: build_initial_monmap 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient: monmap: 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:39.881 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 Processor -- start 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- start start 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient: init 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f1c09c690 0x7f9f1c13c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): picked mon.noname-a con 0x7f9f1c09c690 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1355c0 0x7f9f1c1345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): picked mon.noname-b con 0x7f9f1c1355c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c135ee0 0x7f9f1c134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): picked mon.noname-c con 0x7f9f1c135ee0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c13fd80 con 0x7f9f1c09c690 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c0a2240 con 0x7f9f1c1355c0 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1c13ff00 con 0x7f9f1c135ee0 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f22889640 10 monclient(hunting): authenticate will time out at 1054.061497s 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f1c09c690 0x7f9f1c13c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c135ee0 0x7f9f1c134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f1c09c690 0x7f9f1c13c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53618/0 (socket says 172.21.15.79:53618) 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 1 -- 172.21.15.79:0/3221825482 learned_addr learned my addr 172.21.15.79:0/3221825482 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:39.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): get_auth_request con 0x7f9f1c09c690 auth_method 0 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): get_auth_request con 0x7f9f1c135ee0 auth_method 0 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f137fe640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1355c0 0x7f9f1c1345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1355c0 msgr2=0x7f9f1c1345c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1355c0 0x7f9f1c1345c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f1c09c690 msgr2=0x7f9f1c13c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f1c09c690 0x7f9f1c13c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient: _finish_hunting 0 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 monclient: found mon.noname-c 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f20e26640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c120cb0 con 0x7f9f1c135ee0 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f137fe640 10 monclient: get_auth_request con 0x7f9f1c1355c0 auth_method 0 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f137fe640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1355c0 0x7f9f1c1345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.870+0000 7f9f1bfff640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f1c09c690 0x7f9f1c13c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f20e26640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c135ee0 msgr2=0x7f9f1c134b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f20e26640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c135ee0 0x7f9f1c134b00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9f140ae000 tx=0x7f9f14019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140bc020 con 0x7f9f1c135ee0 2024-04-23T16:41:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: dump: 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 5 monclient: authenticate success, global_id 4400 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f140b8880 con 0x7f9f1c135ee0 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140a5640 con 0x7f9f1c135ee0 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 10 monclient: _renew_subs 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1c071730 con 0x7f9f1c135ee0 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1affd640 4 set_mon_vals no callback set 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 10 monclient: dump: 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 10 monclient: _renew_subs 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1c120f90 con 0x7f9f1c135ee0 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 0x7f9f1c09d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c067f20 con 0x7f9f1c1206f0 2024-04-23T16:41:39.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f9f140bf020 con 0x7f9f1c135ee0 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 1 --2- 172.21.15.79:0/3221825482 >> v2:172.21.15.204:6812/722020384 conn(0x7f9f00036430 0x7f9f000388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 0x7f9f1c09d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f9f140e3dc0 con 0x7f9f1c135ee0 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient: get_auth_request con 0x7f9f1c1206f0 auth_method 0 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f137fe640 1 --2- 172.21.15.79:0/3221825482 >> v2:172.21.15.204:6812/722020384 conn(0x7f9f00036430 0x7f9f000388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f137fe640 10 monclient: get_auth_request con 0x7f9f00036430 auth_method 0 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.874+0000 7f9f1bfff640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f1c0692e0 con 0x7f9f1c1206f0 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f1bfff640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 msgr2=0x7f9f1c09d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f1bfff640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 0x7f9f1c09d2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c011db0 tx=0x7f9f0c00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f137fe640 1 --2- 172.21.15.79:0/3221825482 >> v2:172.21.15.204:6812/722020384 conn(0x7f9f00036430 0x7f9f000388f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c13cb50 tx=0x7f9f04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c004020 con 0x7f9f1c1206f0 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 10 monclient: dump: 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.1 v2:172.21.15.204:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f0c004020 con 0x7f9f1c1206f0 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 msgr2=0x7f9f1c09d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 0x7f9f1c09d2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c011db0 tx=0x7f9f0c00f610 comp rx=0 tx=0).stop 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 msgr2=0x7f9f1c09d2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.878+0000 7f9f197fa640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c1206f0 0x7f9f1c09d2d0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c011db0 tx=0x7f9f0c00f610 comp rx=0 tx=0).stop 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.882+0000 7f9f22889640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.882+0000 7f9f22889640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 0x7f9f1c02c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.882+0000 7f9f22889640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.882+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c139120 con 0x7f9f1c02c450 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 0x7f9f1c02c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient: get_auth_request con 0x7f9f1c02c450 auth_method 0 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 1 -- 172.21.15.79:0/3221825482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f1c06ba20 con 0x7f9f1c02c450 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 msgr2=0x7f9f1c02c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f1bfff640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 0x7f9f1c02c830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c01b000 tx=0x7f9f0c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c003a30 con 0x7f9f1c02c450 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.886+0000 7f9f197fa640 10 monclient: dump: 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:39.897 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 <== mon.1 v2:172.21.15.204:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7f9f0c003a30 con 0x7f9f1c02c450 2024-04-23T16:41:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-23T16:41:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 msgr2=0x7f9f1c02c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 0x7f9f1c02c830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c01b000 tx=0x7f9f0c0046d0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 msgr2=0x7f9f1c02c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f197fa640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f1c02c450 0x7f9f1c02c830 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c01b000 tx=0x7f9f0c0046d0 comp rx=0 tx=0).stop 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "peon", 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: "election_epoch": 14, 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 134694, 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum": [ 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: 0, 2024-04-23T16:41:39.899 INFO:teuthology.orchestra.run.smithi079.stdout: 1, 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: 2 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_age": 2, 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "required_con": "2450521708007866372", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "required_mon": [ 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.900 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_con": "4540701547738038271", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "quorum_mon": [ 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.901 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "outside_quorum": [], 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "extra_probe_peers": [], 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_provider": [], 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": { 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "0469449d-bc82-4171-9959-1bb87fc3dec3", 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "modified": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:41:39.902 INFO:teuthology.orchestra.run.smithi079.stdout: "created": "2024-04-23T16:38:59.932761Z", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release": 19, 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "min_mon_release_name": "squid", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "election_strategy": 1, 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "disallowed_leaders: ": "", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false, 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "tiebreaker_mon": "", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "removed_ranks: ": "", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "features": { 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "persistent": [ 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "kraken", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "luminous", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "mimic", 2024-04-23T16:41:39.903 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap-prune", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "nautilus", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "elector-pinging", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "reef", 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "squid" 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "optional": [] 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "mons": [ 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 0, 2024-04-23T16:41:39.904 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "a", 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3300", 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789", 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.905 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6789/0", 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6789/0", 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 1, 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "b", 2024-04-23T16:41:39.906 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:3300", 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789", 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.204:6789/0", 2024-04-23T16:41:39.907 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.204:6789/0", 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "rank": 2, 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "c", 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrs": { 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:3301", 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790", 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 0 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6790/0", 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "172.21.15.79:6790/0", 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "priority": 0, 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "weight": 0, 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "{}" 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.909 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "feature_map": { 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": [ 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": [ 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.910 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 3 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "client": [ 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 4 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": [ 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "features": "0x3f03cffffffdffff", 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "release": "luminous", 2024-04-23T16:41:39.911 INFO:teuthology.orchestra.run.smithi079.stdout: "num": 1 2024-04-23T16:41:39.912 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-23T16:41:39.912 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-23T16:41:39.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-23T16:41:39.912 INFO:teuthology.orchestra.run.smithi079.stdout: "stretch_mode": false 2024-04-23T16:41:39.912 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:39.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 >> v2:172.21.15.204:6812/722020384 conn(0x7f9f00036430 msgr2=0x7f9f000388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 --2- 172.21.15.79:0/3221825482 >> v2:172.21.15.204:6812/722020384 conn(0x7f9f00036430 0x7f9f000388f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c13cb50 tx=0x7f9f04009040 comp rx=0 tx=0).stop 2024-04-23T16:41:39.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 10 monclient: shutdown 2024-04-23T16:41:39.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c135ee0 msgr2=0x7f9f1c134b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 --2- 172.21.15.79:0/3221825482 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9f1c135ee0 0x7f9f1c134b00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9f140ae000 tx=0x7f9f14019650 comp rx=0 tx=0).stop 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f20e26640 1 -- 172.21.15.79:0/3221825482 reap_dead start 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f20e26640 1 -- 172.21.15.79:0/3221825482 reap_dead start 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 shutdown_connections 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 >> 172.21.15.79:0/3221825482 conn(0x7f9f1c0a6250 msgr2=0x7f9f1c1362f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 shutdown_connections 2024-04-23T16:41:39.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:39.890+0000 7f9f22889640 1 -- 172.21.15.79:0/3221825482 wait complete. 2024-04-23T16:41:39.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-23T16:41:39.914 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-23T16:41:39.915 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-23T16:41:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient: get_monmap_and_config 2024-04-23T16:41:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient: build_initial_monmap 2024-04-23T16:41:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient: monmap: 2024-04-23T16:41:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:40.031799+0000 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:40.031799+0000 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 Processor -- start 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 -- start start 2024-04-23T16:41:40.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient: init 2024-04-23T16:41:40.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:40.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-23T16:41:40.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7df806ca30 0x7f7df80a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): picked mon.noname-a con 0x7f7df806ca30 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7df80a3370 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7df813c8c0 0x7f7df813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): picked mon.noname-c con 0x7f7df813c8c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7df80a2f20 con 0x7f7df80a3370 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f7df80a1b50 con 0x7f7df806ca30 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7df806ce10 con 0x7f7df813c8c0 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7dfdfc6640 10 monclient(hunting): authenticate will time out at 1054.219664s 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7df806ca30 0x7f7df80a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:40.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f7df806ca30 0x7f7df80a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53626/0 (socket says 172.21.15.79:53626) 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 1 -- 172.21.15.79:0/3511731753 learned_addr learned my addr 172.21.15.79:0/3511731753 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 10 monclient(hunting): get_auth_request con 0x7f7df806ca30 auth_method 0 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.026+0000 7f7df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:40.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df7fff640 1 --2- 172.21.15.79:0/3511731753 >> v2:172.21.15.204:3300/0 conn(0x7f7df813c8c0 0x7f7df813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df7fff640 10 monclient(hunting): get_auth_request con 0x7f7df813c8c0 auth_method 0 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df7fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 10 monclient(hunting): handle_auth_done global_id 14124 payload 274 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 1 -- 172.21.15.79:0/3511731753 >> v2:172.21.15.204:3300/0 conn(0x7f7df813c8c0 msgr2=0x7f7df813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 1 --2- 172.21.15.79:0/3511731753 >> v2:172.21.15.204:3300/0 conn(0x7f7df813c8c0 0x7f7df813ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 1 -- 172.21.15.79:0/3511731753 >> v1:172.21.15.204:6789/0 conn(0x7f7df80a3370 legacy=0x7f7df80a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 1 monclient: found mon.noname-a 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:40.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.030+0000 7f7df77fe640 1 -- 172.21.15.79:0/3511731753 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df80a5860 con 0x7f7df806ca30 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df77fe640 1 -- 172.21.15.79:0/3511731753 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 msgr2=0x7f7df80a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df77fe640 1 --2- 172.21.15.79:0/3511731753 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df80a2830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7df80a7230 tx=0x7f7de801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bf99af1509c134 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 1 -- 172.21.15.79:0/3511731753 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de80b0070 con 0x7f7df806ca30 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: dump: 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: _finish_auth 0 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: _check_auth_tickets 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 1 -- 172.21.15.79:0/3511731753 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de80ab600 con 0x7f7df806ca30 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 5 monclient: authenticate success, global_id 14124 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 1 -- 172.21.15.79:0/3511731753 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de80ab9d0 con 0x7f7df806ca30 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 4 set_mon_vals no callback set 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7df67fc640 10 monclient: dump: 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:40.041 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:40.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 10 monclient: shutdown 2024-04-23T16:41:40.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:40.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3511731753 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 msgr2=0x7f7df80a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:40.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3511731753 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df80a2830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7df80a7230 tx=0x7f7de801c430 comp rx=0 tx=0).stop 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3511731753 shutdown_connections 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3511731753 >> v2:172.21.15.204:3300/0 conn(0x7f7df813c8c0 0x7f7df813ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3511731753 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df80a2830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3511731753 >> 172.21.15.79:0/3511731753 conn(0x7f7df80aa3e0 msgr2=0x7f7df80a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3511731753 shutdown_connections 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3511731753 wait complete. 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 10 monclient: build_initial_monmap 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.034+0000 7f7dfdfc6640 10 monclient: monmap: 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:40.047 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 Processor -- start 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 -- start start 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: init 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df809f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): picked mon.noname-a con 0x7f7df806ca30 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:40.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7df80a3370 0x7f7df8120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): picked mon.noname-c con 0x7f7df80a3370 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df813c8c0 0x7f7df8138850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): picked mon.noname-b con 0x7f7df813c8c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8141da0 con 0x7f7df806ca30 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8141c20 con 0x7f7df813c8c0 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7df8141f20 con 0x7f7df80a3370 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient(hunting): authenticate will time out at 1054.228625s 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df809f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:40.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df809f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53640/0 (socket says 172.21.15.79:53640) 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 -- 172.21.15.79:0/3052171004 learned_addr learned my addr 172.21.15.79:0/3052171004 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): get_auth_request con 0x7f7df806ca30 auth_method 0 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7df80a3370 0x7f7df8120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): get_auth_request con 0x7f7df80a3370 auth_method 0 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:40.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient(hunting): handle_auth_done global_id 14127 payload 824 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 -- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7df80a3370 msgr2=0x7f7df8120430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7df80a3370 0x7f7df8120430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 -- 172.21.15.79:0/3052171004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df813c8c0 msgr2=0x7f7df8138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df813c8c0 0x7f7df8138850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 monclient: found mon.noname-a 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:40.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 -- 172.21.15.79:0/3052171004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df813dc20 con 0x7f7df806ca30 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 -- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 msgr2=0x7f7df809f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df77fe640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df809f900 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7de80ae040 tx=0x7f7de801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de80ba020 con 0x7f7df806ca30 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: dump: 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:40.052 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: _finish_auth 0 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: _check_auth_tickets 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de80b6830 con 0x7f7df806ca30 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 5 monclient: authenticate success, global_id 14127 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: _renew_subs 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df8121ff0 con 0x7f7df806ca30 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de80b0040 con 0x7f7df806ca30 2024-04-23T16:41:40.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7ddffff640 10 monclient: dump: 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: _renew_subs 2024-04-23T16:41:40.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df8122940 con 0x7f7df806ca30 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.038+0000 7f7df5ffb640 4 set_mon_vals no callback set 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f7de80bd020 con 0x7f7df806ca30 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7dfdfc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7dfdfc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7dfdfc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df8068ae0 con 0x7f7df806ca30 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7ddffff640 1 --2- 172.21.15.79:0/3052171004 >> v2:172.21.15.204:6812/722020384 conn(0x7f7de4036430 0x7f7de40388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f7de80e3330 con 0x7f7df806ca30 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7df6ffd640 1 --2- 172.21.15.79:0/3052171004 >> v2:172.21.15.204:6812/722020384 conn(0x7f7de4036430 0x7f7de40388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:40.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7df6ffd640 10 monclient: get_auth_request con 0x7f7de4036430 auth_method 0 2024-04-23T16:41:40.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.042+0000 7f7df6ffd640 1 --2- 172.21.15.79:0/3052171004 >> v2:172.21.15.204:6812/722020384 conn(0x7f7de4036430 0x7f7de40388f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7df80a0070 tx=0x7f7dec007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:40.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.046+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7de80c4030 con 0x7f7df806ca30 2024-04-23T16:41:40.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.046+0000 7f7ddffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:40.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.046+0000 7f7ddffff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7dfdfc6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-23T16:41:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7dfdfc6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-23T16:41:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7dfdfc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7df8072040 con 0x7f7df806ca30 2024-04-23T16:41:40.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7ddffff640 1 -- 172.21.15.79:0/3052171004 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7de80160b0 con 0x7f7df806ca30 2024-04-23T16:41:40.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7ddffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-23T16:41:40.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f7ddffff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:40.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=1198068891,config=692619927,health=3897446751,kv=3804351316,logm=1243972517}) 2024-04-23T16:41:40.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=11,logm=67} crc {auth=2984416250,config=692619927,health=3897446751,kv=281921197,logm=2453705513}) 2024-04-23T16:41:40.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=1198068891,config=692619927,health=3897446751,kv=3804351316,logm=1243972517}) 2024-04-23T16:41:40.276 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.270+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=7,config=2,health=11,kv=11,logm=69} crc {auth=2408127713,config=692619927,health=1625394897,kv=281921197,logm=3119664261}) 2024-04-23T16:41:40.277 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.278 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440650403}) 2024-04-23T16:41:40.278 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3994346076}) 2024-04-23T16:41:40.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 >> v2:172.21.15.204:6812/722020384 conn(0x7f7de4036430 msgr2=0x7f7de40388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:40.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3052171004 >> v2:172.21.15.204:6812/722020384 conn(0x7f7de4036430 0x7f7de40388f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7df80a0070 tx=0x7f7dec007e40 comp rx=0 tx=0).stop 2024-04-23T16:41:40.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 10 monclient: shutdown 2024-04-23T16:41:40.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:40.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 msgr2=0x7f7df809f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:40.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df809f900 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7de80ae040 tx=0x7f7de801ac70 comp rx=0 tx=0).stop 2024-04-23T16:41:40.279 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.279 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440650403}) 2024-04-23T16:41:40.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 shutdown_connections 2024-04-23T16:41:40.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3052171004 >> v2:172.21.15.204:6812/722020384 conn(0x7f7de4036430 0x7f7de40388f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.279 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3589711497}) 2024-04-23T16:41:40.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7df813c8c0 0x7f7df8138850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7df80a3370 0x7f7df8120430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 --2- 172.21.15.79:0/3052171004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7df806ca30 0x7f7df809f900 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:40.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 >> 172.21.15.79:0/3052171004 conn(0x7f7df80aa3e0 msgr2=0x7f7df813d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:40.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 shutdown_connections 2024-04-23T16:41:40.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f7dfdfc6640 1 -- 172.21.15.79:0/3052171004 wait complete. 2024-04-23T16:41:40.281 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.281 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152374049}) 2024-04-23T16:41:40.281 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2640104621}) 2024-04-23T16:41:40.281 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.281 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152374049}) 2024-04-23T16:41:40.281 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.274+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017484933}) 2024-04-23T16:41:40.282 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.282 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109352679}) 2024-04-23T16:41:40.282 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641155737}) 2024-04-23T16:41:40.283 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.283 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109352679}) 2024-04-23T16:41:40.283 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1291292099}) 2024-04-23T16:41:40.285 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.278+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.285 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=53,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=3} crc {logm=1406859024,mdsmap=2091496210,mgr=2470432358,mgr_command_descs=3636332126,mgr_metadata=2990048560,mgrstat=2974797077,mon_config_key=2419584356,monmap=3140514009,osd_metadata=861717671,osd_pg_creating=1344444403,osdmap=973863822}) 2024-04-23T16:41:40.285 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=54,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=3} crc {logm=2373337698,mdsmap=2840952586,mgr=3881277981,mgr_command_descs=3636332126,mgr_metadata=2990048560,mgrstat=2974797077,mon_config_key=2419584356,monmap=3706722640,osd_metadata=861717671,osd_pg_creating=1344444403,osdmap=973863822}) 2024-04-23T16:41:40.285 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.285 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=53,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=3} crc {logm=1406859024,mdsmap=2091496210,mgr=2470432358,mgr_command_descs=3636332126,mgr_metadata=2990048560,mgrstat=2974797077,mon_config_key=2419584356,monmap=3140514009,osd_metadata=861717671,osd_pg_creating=1344444403,osdmap=973863822}) 2024-04-23T16:41:40.285 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=53,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osdmap=5} crc {logm=1815137101,mdsmap=2840952586,mgr=2470432359,mgr_command_descs=3636332126,mgr_metadata=2990048560,mgrstat=1128513771,mon_config_key=3930501236,monmap=3375904018,osd_metadata=2376871847,osd_pg_creating=1344444403,osdmap=3971755916}) 2024-04-23T16:41:40.286 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.286 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=2259683089}) 2024-04-23T16:41:40.286 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=40} crc {osdmap=2811656319}) 2024-04-23T16:41:40.286 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-23T16:41:40.287 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=2259683089}) 2024-04-23T16:41:40.287 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-23T16:41:40.282+0000 7f8553ea7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=41} crc {osdmap=683859346}) 2024-04-23T16:41:40.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-23T16:41:41.297 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-23T16:41:41.297 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-23T16:41:41.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient: get_monmap_and_config 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient: build_initial_monmap 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient: monmap: 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:41.426670+0000 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:41.426670+0000 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:41.427 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:41.427 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:41.427 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:41.427 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 Processor -- start 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 -- start start 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient: init 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fda84135a20 0x7fda84135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): picked mon.noname-c con 0x7fda84135a20 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 0x7fda8409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): picked mon.noname-b con 0x7fda8409f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda84135300 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda84072f80 con 0x7fda84135300 2024-04-23T16:41:41.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fda8409fb10 con 0x7fda84135a20 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fda8409fdf0 con 0x7fda8409f430 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8c05a640 10 monclient(hunting): authenticate will time out at 1055.614517s 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 0x7fda8409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8a5f7640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fda84135300 0x7fda8409bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33398/0 (socket says 172.21.15.79:33398) 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda8a5f7640 1 -- 172.21.15.79:0/2366259882 learned_addr learned my addr 172.21.15.79:0/2366259882 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): get_auth_request con 0x7fda8409f430 auth_method 0 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 1 -- 172.21.15.79:0/2366259882 >> v2:172.21.15.204:3300/0 conn(0x7fda84135a20 msgr2=0x7fda84135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 1 --2- 172.21.15.79:0/2366259882 >> v2:172.21.15.204:3300/0 conn(0x7fda84135a20 0x7fda84135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 1 -- 172.21.15.79:0/2366259882 >> v1:172.21.15.79:6789/0 conn(0x7fda84135300 legacy=0x7fda8409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient: _finish_hunting 0 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 1 monclient: found mon.noname-b 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.422+0000 7fda895f5640 1 -- 172.21.15.79:0/2366259882 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8406de30 con 0x7fda8409f430 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda895f5640 1 -- 172.21.15.79:0/2366259882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 msgr2=0x7fda8409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda895f5640 1 --2- 172.21.15.79:0/2366259882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 0x7fda8409ba00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fda780097c0 tx=0x7fda7801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc175bcaf2bccb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 1 -- 172.21.15.79:0/2366259882 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780b0070 con 0x7fda8409f430 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: dump: 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.436 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: _finish_auth 0 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: _check_auth_tickets 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 5 monclient: authenticate success, global_id 4406 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 1 -- 172.21.15.79:0/2366259882 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda780ab600 con 0x7fda8409f430 2024-04-23T16:41:41.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 4 set_mon_vals no callback set 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 1 -- 172.21.15.79:0/2366259882 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780ab9d0 con 0x7fda8409f430 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda88df4640 10 monclient: dump: 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.438 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 10 monclient: shutdown 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 1 -- 172.21.15.79:0/2366259882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 msgr2=0x7fda8409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.426+0000 7fda8c05a640 1 --2- 172.21.15.79:0/2366259882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 0x7fda8409ba00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fda780097c0 tx=0x7fda7801c320 comp rx=0 tx=0).stop 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- 172.21.15.79:0/2366259882 shutdown_connections 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 --2- 172.21.15.79:0/2366259882 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda8409f430 0x7fda8409ba00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 --2- 172.21.15.79:0/2366259882 >> v2:172.21.15.204:3300/0 conn(0x7fda84135a20 0x7fda84135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- 172.21.15.79:0/2366259882 >> 172.21.15.79:0/2366259882 conn(0x7fda840a25b0 msgr2=0x7fda840a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- 172.21.15.79:0/2366259882 shutdown_connections 2024-04-23T16:41:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- 172.21.15.79:0/2366259882 wait complete. 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient: build_initial_monmap 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient: monmap: 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 Processor -- start 2024-04-23T16:41:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- start start 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient: init 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8409f430 0x7fda8413c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): picked mon.noname-b con 0x7fda8409f430 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 0x7fda841342e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): picked mon.noname-c con 0x7fda84135300 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda84135a20 0x7fda841cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): picked mon.noname-a con 0x7fda84135a20 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fda841400b0 con 0x7fda84135a20 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda8409c860 con 0x7fda8409f430 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fda84140230 con 0x7fda84135300 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8c05a640 10 monclient(hunting): authenticate will time out at 1055.621543s 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 0x7fda841342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 0x7fda841342e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45466/0 (socket says 172.21.15.79:45466) 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 -- 172.21.15.79:0/481321444 learned_addr learned my addr 172.21.15.79:0/481321444 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): get_auth_request con 0x7fda84135300 auth_method 0 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda84135a20 0x7fda841cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): get_auth_request con 0x7fda84135a20 auth_method 0 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda8a5f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda89df6640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8409f430 0x7fda8413c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 -- 172.21.15.79:0/481321444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8409f430 msgr2=0x7fda8413c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8409f430 0x7fda8413c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 -- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda84135a20 msgr2=0x7fda841cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda84135a20 0x7fda841cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient: _finish_hunting 0 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 monclient: found mon.noname-c 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.430+0000 7fda895f5640 1 -- 172.21.15.79:0/481321444 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8411ec70 con 0x7fda84135300 2024-04-23T16:41:41.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda895f5640 1 -- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 msgr2=0x7fda841342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:41.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda895f5640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 0x7fda841342e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fda780097c0 tx=0x7fda780190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:41.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780b8020 con 0x7fda84135300 2024-04-23T16:41:41.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: dump: 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: _finish_auth 0 2024-04-23T16:41:41.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: _check_auth_tickets 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 5 monclient: authenticate success, global_id 4409 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda780b4810 con 0x7fda84135300 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 monclient: _renew_subs 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780b0040 con 0x7fda84135300 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda84071aa0 con 0x7fda84135300 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 10 monclient: dump: 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.448 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda77fff640 4 set_mon_vals no callback set 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 monclient: _renew_subs 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8411e990 con 0x7fda84135300 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda840692e0 con 0x7fda84135300 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fda780bb020 con 0x7fda84135300 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 1 --2- 172.21.15.79:0/481321444 >> v2:172.21.15.204:6812/722020384 conn(0x7fda680363e0 0x7fda680388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fda780e2d20 con 0x7fda84135300 2024-04-23T16:41:41.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda89df6640 1 --2- 172.21.15.79:0/481321444 >> v2:172.21.15.204:6812/722020384 conn(0x7fda680363e0 0x7fda680388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.434+0000 7fda89df6640 10 monclient: get_auth_request con 0x7fda680363e0 auth_method 0 2024-04-23T16:41:41.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.438+0000 7fda89df6640 1 --2- 172.21.15.79:0/481321444 >> v2:172.21.15.204:6812/722020384 conn(0x7fda680363e0 0x7fda680388a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda8000a830 tx=0x7fda80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:41.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.438+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fda780c2030 con 0x7fda84135300 2024-04-23T16:41:41.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.438+0000 7fda767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:41.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.438+0000 7fda767fc640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:41.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.662+0000 7fda8c05a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-23T16:41:41.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.662+0000 7fda8c05a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-23T16:41:41.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.662+0000 7fda8c05a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.662+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda8411d260 con 0x7fda84135300 2024-04-23T16:41:41.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda767fc640 1 -- 172.21.15.79:0/481321444 <== mon.2 v2:172.21.15.79:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fda78016030 con 0x7fda84135300 2024-04-23T16:41:41.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-23T16:41:41.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda767fc640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:41.669 INFO:teuthology.orchestra.run.smithi079.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","modified":"2024-04-23T16:38:59.932761Z","created":"2024-04-23T16:38:59.932761Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 >> v2:172.21.15.204:6812/722020384 conn(0x7fda680363e0 msgr2=0x7fda680388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 --2- 172.21.15.79:0/481321444 >> v2:172.21.15.204:6812/722020384 conn(0x7fda680363e0 0x7fda680388a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda8000a830 tx=0x7fda80008040 comp rx=0 tx=0).stop 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 10 monclient: shutdown 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 msgr2=0x7fda841342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 0x7fda841342e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fda780097c0 tx=0x7fda780190c0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 shutdown_connections 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 --2- 172.21.15.79:0/481321444 >> v2:172.21.15.204:6812/722020384 conn(0x7fda680363e0 0x7fda680388a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda84135a20 0x7fda841cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fda84135300 0x7fda841342e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 --2- 172.21.15.79:0/481321444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8409f430 0x7fda8413c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 >> 172.21.15.79:0/481321444 conn(0x7fda840a25b0 msgr2=0x7fda84136400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:41.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 shutdown_connections 2024-04-23T16:41:41.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.666+0000 7fda8c05a640 1 -- 172.21.15.79:0/481321444 wait complete. 2024-04-23T16:41:41.687 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-23T16:41:41.687 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-23T16:41:41.701 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T16:41:41.701 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient: get_monmap_and_config 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient: build_initial_monmap 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient: monmap: 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:41.810990+0000 2024-04-23T16:41:41.810 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:41.810990+0000 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:41.811 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 Processor -- start 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 -- start start 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient: init 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b50135a20 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 0x7f5b50135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): picked mon.noname-b con 0x7f5b5009f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 0x7f5b5009c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): picked mon.noname-a con 0x7f5b50135300 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:41.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b50072f80 con 0x7f5b50135a20 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f5b5009fb10 con 0x7f5b50135300 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5b5009fdf0 con 0x7f5b5009f430 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b56fda640 10 monclient(hunting): authenticate will time out at 1055.998791s 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 0x7f5b50135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b54d76640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f5b50135a20 0x7f5b5009ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33406/0 (socket says 172.21.15.79:33406) 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b54d76640 1 -- 172.21.15.79:0/2436903802 learned_addr learned my addr 172.21.15.79:0/2436903802 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b55577640 1 --2- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 0x7f5b5009c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): get_auth_request con 0x7f5b5009f430 auth_method 0 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b55577640 1 -- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 msgr2=0x7f5b5009c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b55577640 1 -- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 msgr2=0x7f5b5009c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b55577640 1 --2- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 0x7f5b5009c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b55577640 1 --2- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 0x7f5b5009c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 1 -- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 msgr2=0x7f5b5009c110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 1 --2- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 0x7f5b5009c110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 1 -- 172.21.15.79:0/2436903802 >> v1:172.21.15.79:6789/0 conn(0x7f5b50135a20 legacy=0x7f5b5009ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 1 monclient: found mon.noname-b 2024-04-23T16:41:41.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:41.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:41.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.806+0000 7f5b47fff640 1 -- 172.21.15.79:0/2436903802 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5006de30 con 0x7f5b5009f430 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b47fff640 1 -- 172.21.15.79:0/2436903802 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 msgr2=0x7f5b50135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b47fff640 1 --2- 172.21.15.79:0/2436903802 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 0x7f5b50135e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097c0 tx=0x7f5b4c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68c9dbcf95d66c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 1 -- 172.21.15.79:0/2436903802 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c0b0070 con 0x7f5b5009f430 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: dump: 2024-04-23T16:41:41.819 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:41.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 1 -- 172.21.15.79:0/2436903802 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4c0ab600 con 0x7f5b5009f430 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b56fda640 5 monclient: authenticate success, global_id 4412 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 1 -- 172.21.15.79:0/2436903802 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c0ab9d0 con 0x7f5b5009f430 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b477fe640 10 monclient: dump: 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.821 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b56fda640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.810+0000 7f5b56fda640 4 set_mon_vals no callback set 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient: shutdown 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- 172.21.15.79:0/2436903802 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 msgr2=0x7f5b50135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 --2- 172.21.15.79:0/2436903802 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 0x7f5b50135e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097c0 tx=0x7f5b4c01c320 comp rx=0 tx=0).stop 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- 172.21.15.79:0/2436903802 shutdown_connections 2024-04-23T16:41:41.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 --2- 172.21.15.79:0/2436903802 >> v2:172.21.15.79:3300/0 conn(0x7f5b50135300 0x7f5b5009c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 --2- 172.21.15.79:0/2436903802 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b5009f430 0x7f5b50135e00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- 172.21.15.79:0/2436903802 >> 172.21.15.79:0/2436903802 conn(0x7f5b500a25b0 msgr2=0x7f5b500a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- 172.21.15.79:0/2436903802 shutdown_connections 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- 172.21.15.79:0/2436903802 wait complete. 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient: build_initial_monmap 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient: monmap: 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:41.823 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 Processor -- start 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- start start 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient: init 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 0x7f5b50133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): picked mon.noname-a con 0x7f5b5009f430 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 0x7f5b50133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): picked mon.noname-c con 0x7f5b50135300 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b50135a20 0x7f5b501cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): picked mon.noname-b con 0x7f5b50135a20 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501400b0 con 0x7f5b5009f430 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5009c860 con 0x7f5b50135a20 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5b50140230 con 0x7f5b50135300 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b56fda640 10 monclient(hunting): authenticate will time out at 1056.005577s 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 0x7f5b50133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 0x7f5b50133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 0x7f5b50133bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45494/0 (socket says 172.21.15.79:45494) 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 0x7f5b50133680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53674/0 (socket says 172.21.15.79:53674) 2024-04-23T16:41:41.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 1 -- 172.21.15.79:0/384988429 learned_addr learned my addr 172.21.15.79:0/384988429 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): get_auth_request con 0x7f5b50135300 auth_method 0 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): get_auth_request con 0x7f5b5009f430 auth_method 0 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:41.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b55577640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b50135a20 0x7f5b501cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b55577640 10 monclient(hunting): get_auth_request con 0x7f5b50135a20 auth_method 0 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b55577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b55577640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b55577640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient(hunting): handle_auth_done global_id 14133 payload 824 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 -- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 msgr2=0x7f5b50133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 0x7f5b50133bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 -- 172.21.15.79:0/384988429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b50135a20 msgr2=0x7f5b501cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b50135a20 0x7f5b501cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient: _finish_hunting 0 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 monclient: found mon.noname-a 2024-04-23T16:41:41.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:41.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:41.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b54d76640 1 -- 172.21.15.79:0/384988429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5011dd60 con 0x7f5b5009f430 2024-04-23T16:41:41.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.814+0000 7f5b47fff640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 0x7f5b50133bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:41.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b54d76640 1 -- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 msgr2=0x7f5b50133680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:41.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b54d76640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 0x7f5b50133680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000c410 tx=0x7f5b4000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:41.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40017070 con 0x7f5b5009f430 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: dump: 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.830 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 5 monclient: authenticate success, global_id 14133 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 monclient: _renew_subs 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b400070d0 con 0x7f5b5009f430 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b50072c70 con 0x7f5b5009f430 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b400074c0 con 0x7f5b5009f430 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:41.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 10 monclient: dump: 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 monclient: _renew_subs 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b46ffd640 4 set_mon_vals no callback set 2024-04-23T16:41:41.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b5009e410 con 0x7f5b5009f430 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f5b4001a070 con 0x7f5b5009f430 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b5011d5d0 con 0x7f5b5009f430 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 1 --2- 172.21.15.79:0/384988429 >> v2:172.21.15.204:6812/722020384 conn(0x7f5b3c0363e0 0x7f5b3c0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f5b40048430 con 0x7f5b5009f430 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b47fff640 1 --2- 172.21.15.79:0/384988429 >> v2:172.21.15.204:6812/722020384 conn(0x7f5b3c0363e0 0x7f5b3c0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.818+0000 7f5b47fff640 10 monclient: get_auth_request con 0x7f5b3c0363e0 auth_method 0 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.822+0000 7f5b47fff640 1 --2- 172.21.15.79:0/384988429 >> v2:172.21.15.204:6812/722020384 conn(0x7f5b3c0363e0 0x7f5b3c0388a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c01cbc0 tx=0x7f5b4c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.822+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5b40010300 con 0x7f5b5009f430 2024-04-23T16:41:41.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.822+0000 7f5b457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:41.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.822+0000 7f5b457fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:41.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.978+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b500692e0 con 0x7f5b3c0363e0 2024-04-23T16:41:41.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b457fa640 1 -- 172.21.15.79:0/384988429 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33808 (secure 0 0 0) 0x7f5b500692e0 con 0x7f5b3c0363e0 2024-04-23T16:41:41.986 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:41:41.986 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:41:41.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 >> v2:172.21.15.204:6812/722020384 conn(0x7f5b3c0363e0 msgr2=0x7f5b3c0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 --2- 172.21.15.79:0/384988429 >> v2:172.21.15.204:6812/722020384 conn(0x7f5b3c0363e0 0x7f5b3c0388a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c01cbc0 tx=0x7f5b4c008040 comp rx=0 tx=0).stop 2024-04-23T16:41:41.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 10 monclient: shutdown 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 msgr2=0x7f5b50133680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 0x7f5b50133680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000c410 tx=0x7f5b4000c8d0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 shutdown_connections 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 --2- 172.21.15.79:0/384988429 >> v2:172.21.15.204:6812/722020384 conn(0x7f5b3c0363e0 0x7f5b3c0388a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b50135a20 0x7f5b501cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b50135300 0x7f5b50133bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 --2- 172.21.15.79:0/384988429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b5009f430 0x7f5b50133680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 >> 172.21.15.79:0/384988429 conn(0x7f5b500a25b0 msgr2=0x7f5b50134490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 shutdown_connections 2024-04-23T16:41:41.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:41.982+0000 7f5b56fda640 1 -- 172.21.15.79:0/384988429 wait complete. 2024-04-23T16:41:42.003 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-23T16:41:41.331042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163328,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067712,"statfs":{"total":579820584960,"available":579653337088,"internally_reserved":0,"allocated":1236992,"data_stored":1440242,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.909224+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:40:08.909224+0000","last_peered":"2024-04-23T16:40:08.909224+0000","last_clean":"2024-04-23T16:40:08.909224+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:40:08.909224+0000","last_undegraded":"2024-04-23T16:40:08.909224+0000","last_fullsized":"2024-04-23T16:40:08.909224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.031042+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:40:09.031042+0000","last_peered":"2024-04-23T16:40:09.031042+0000","last_clean":"2024-04-23T16:40:09.031042+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:40:09.031042+0000","last_undegraded":"2024-04-23T16:40:09.031042+0000","last_fullsized":"2024-04-23T16:40:09.031042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.908626+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:40:08.908626+0000","last_peered":"2024-04-23T16:40:08.908626+0000","last_clean":"2024-04-23T16:40:08.908626+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:40:08.908626+0000","last_undegraded":"2024-04-23T16:40:08.908626+0000","last_fullsized":"2024-04-23T16:40:08.908626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.006900+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:40:09.006900+0000","last_peered":"2024-04-23T16:40:09.006900+0000","last_clean":"2024-04-23T16:40:09.006900+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:40:09.006900+0000","last_undegraded":"2024-04-23T16:40:09.006900+0000","last_fullsized":"2024-04-23T16:40:09.006900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.039969+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:40:09.039969+0000","last_peered":"2024-04-23T16:40:09.039969+0000","last_clean":"2024-04-23T16:40:09.039969+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:40:09.039969+0000","last_undegraded":"2024-04-23T16:40:09.039969+0000","last_fullsized":"2024-04-23T16:40:09.039969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.905773+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:40:08.905773+0000","last_peered":"2024-04-23T16:40:08.905773+0000","last_clean":"2024-04-23T16:40:08.905773+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:40:08.905773+0000","last_undegraded":"2024-04-23T16:40:08.905773+0000","last_fullsized":"2024-04-23T16:40:08.905773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.032667+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:40:09.032667+0000","last_peered":"2024-04-23T16:40:09.032667+0000","last_clean":"2024-04-23T16:40:09.032667+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:40:09.032667+0000","last_undegraded":"2024-04-23T16:40:09.032667+0000","last_fullsized":"2024-04-23T16:40:09.032667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.906573+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:40:08.906573+0000","last_peered":"2024-04-23T16:40:08.906573+0000","last_clean":"2024-04-23T16:40:08.906573+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:40:08.906573+0000","last_undegraded":"2024-04-23T16:40:08.906573+0000","last_fullsized":"2024-04-23T16:40:08.906573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":175,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.907848+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:40:08.907848+0000","last_peered":"2024-04-23T16:40:08.907848+0000","last_clean":"2024-04-23T16:40:08.907848+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:40:08.907848+0000","last_undegraded":"2024-04-23T16:40:08.907848+0000","last_fullsized":"2024-04-23T16:40:08.907848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607579,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:41:42.004 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:41:42.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient: get_monmap_and_config 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient: build_initial_monmap 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient: monmap: 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:42.111750+0000 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:42.111750+0000 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.111 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:42.112 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:42.112 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:42.112 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:42.112 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:42.112 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 Processor -- start 2024-04-23T16:41:42.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 -- start start 2024-04-23T16:41:42.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient: init 2024-04-23T16:41:42.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:42.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-23T16:41:42.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea4135a20 0x7f9ea4135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9ea4135a20 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f9ea409f430 0x7f9ea409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea409f430 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9ea4135300 0x7f9ea409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): picked mon.noname-c con 0x7f9ea4135300 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea4072f80 con 0x7f9ea409f430 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea409fb10 con 0x7f9ea4135300 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea409fdf0 con 0x7f9ea4135a20 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:42.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9eaae5c640 10 monclient(hunting): authenticate will time out at 1056.299801s 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea8bf8640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea4135a20 0x7f9ea4135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea3fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f9ea409f430 0x7f9ea409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea3fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f9ea409f430 0x7f9ea409ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53684/0 (socket says 172.21.15.79:53684) 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea3fff640 1 -- 172.21.15.79:0/3891411238 learned_addr learned my addr 172.21.15.79:0/3891411238 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea8bf8640 10 monclient(hunting): get_auth_request con 0x7f9ea4135a20 auth_method 0 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea8bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea8bf8640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.106+0000 7f9ea8bf8640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): get_auth_request con 0x7f9ea409f430 auth_method 0 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_done global_id 14136 payload 274 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 1 -- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea4135a20 msgr2=0x7f9ea4135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 1 --2- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea4135a20 0x7f9ea4135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea93f9640 1 --2- 172.21.15.79:0/3891411238 >> v2:172.21.15.204:3300/0 conn(0x7f9ea4135300 0x7f9ea409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 1 -- 172.21.15.79:0/3891411238 >> v2:172.21.15.204:3300/0 conn(0x7f9ea4135300 msgr2=0x7f9ea409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:42.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 1 --2- 172.21.15.79:0/3891411238 >> v2:172.21.15.204:3300/0 conn(0x7f9ea4135300 0x7f9ea409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:42.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 1 monclient: found mon.noname-a 2024-04-23T16:41:42.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:42.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.110+0000 7f9ea3fff640 1 -- 172.21.15.79:0/3891411238 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea406de30 con 0x7f9ea409f430 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 1 -- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea409f430 msgr2=0x7f9ea409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 1 --2- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea409f430 0x7f9ea409ba00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4136400 tx=0x7f9e8c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a624490817deebec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 1 -- 172.21.15.79:0/3891411238 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c0a5530 con 0x7f9ea409f430 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: dump: 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.123 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 1 -- 172.21.15.79:0/3891411238 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e8c0a56d0 con 0x7f9ea409f430 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 1 -- 172.21.15.79:0/3891411238 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c0a5aa0 con 0x7f9ea409f430 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 5 monclient: authenticate success, global_id 14136 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:42.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 4 set_mon_vals no callback set 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea37fe640 10 monclient: dump: 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: shutdown 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea409f430 msgr2=0x7f9ea409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea409f430 0x7f9ea409ba00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4136400 tx=0x7f9e8c01c8b0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- 172.21.15.79:0/3891411238 shutdown_connections 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/3891411238 >> v2:172.21.15.204:3300/0 conn(0x7f9ea4135300 0x7f9ea409bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea409f430 0x7f9ea409ba00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/3891411238 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea4135a20 0x7f9ea4135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- 172.21.15.79:0/3891411238 >> 172.21.15.79:0/3891411238 conn(0x7f9ea40a25b0 msgr2=0x7f9ea40a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- 172.21.15.79:0/3891411238 shutdown_connections 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- 172.21.15.79:0/3891411238 wait complete. 2024-04-23T16:41:42.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: build_initial_monmap 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: monmap: 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 Processor -- start 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- start start 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: init 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 0x7f9ea41cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): picked mon.noname-c con 0x7f9ea409f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea4135300 0x7f9ea41345a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea4135300 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ea4135a20 0x7f9ea4134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9ea4135a20 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea413f0a0 con 0x7f9ea4135300 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea409c890 con 0x7f9ea4135a20 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea413f220 con 0x7f9ea409f430 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9eaae5c640 10 monclient(hunting): authenticate will time out at 1056.306762s 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 0x7f9ea41cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 0x7f9ea41cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45518/0 (socket says 172.21.15.79:45518) 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 -- 172.21.15.79:0/510071511 learned_addr learned my addr 172.21.15.79:0/510071511 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): get_auth_request con 0x7f9ea409f430 auth_method 0 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea4135300 0x7f9ea41345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): get_auth_request con 0x7f9ea4135300 auth_method 0 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-04-23T16:41:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 -- 172.21.15.79:0/510071511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ea4135a20 msgr2=0x7f9ea4134ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ea4135a20 0x7f9ea4134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 -- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea4135300 msgr2=0x7f9ea41345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea4135300 0x7f9ea41345a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient: _finish_hunting 0 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 monclient: found mon.noname-c 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.114+0000 7f9ea8bf8640 1 -- 172.21.15.79:0/510071511 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea4072c70 con 0x7f9ea409f430 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea8bf8640 1 -- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 msgr2=0x7f9ea41cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea8bf8640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 0x7f9ea41cd530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400e9b0 tx=0x7f9e9400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94017070 con 0x7f9ea409f430 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: dump: 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e9400cbb0 con 0x7f9ea409f430 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94013470 con 0x7f9ea409f430 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 10 monclient: dump: 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 5 monclient: authenticate success, global_id 4421 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 monclient: _renew_subs 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea2ffd640 4 set_mon_vals no callback set 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea4071b90 con 0x7f9ea409f430 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 monclient: _renew_subs 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea411e530 con 0x7f9ea409f430 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea40692e0 con 0x7f9ea409f430 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f9e9401a070 con 0x7f9ea409f430 2024-04-23T16:41:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 1 --2- 172.21.15.79:0/510071511 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e840363e0 0x7f9e840388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f9e94049060 con 0x7f9ea409f430 2024-04-23T16:41:42.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea3fff640 1 --2- 172.21.15.79:0/510071511 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e840363e0 0x7f9e840388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.118+0000 7f9ea3fff640 10 monclient: get_auth_request con 0x7f9e840363e0 auth_method 0 2024-04-23T16:41:42.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.122+0000 7f9ea3fff640 1 --2- 172.21.15.79:0/510071511 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e840363e0 0x7f9e840388a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ea406cba0 tx=0x7f9e8c0a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.122+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9e94013e10 con 0x7f9ea409f430 2024-04-23T16:41:42.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.122+0000 7f9ea17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.122+0000 7f9ea17fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:42.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.270+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ea411dcb0 con 0x7f9e840363e0 2024-04-23T16:41:42.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9ea17fa640 1 -- 172.21.15.79:0/510071511 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33808 (secure 0 0 0) 0x7f9e8c009150 con 0x7f9e840363e0 2024-04-23T16:41:42.278 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:41:42.278 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e840363e0 msgr2=0x7f9e840388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/510071511 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e840363e0 0x7f9e840388a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ea406cba0 tx=0x7f9e8c0a7070 comp rx=0 tx=0).stop 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 10 monclient: shutdown 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 msgr2=0x7f9ea41cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 0x7f9ea41cd530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400e9b0 tx=0x7f9e9400ee70 comp rx=0 tx=0).stop 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 shutdown_connections 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/510071511 >> v2:172.21.15.204:6812/722020384 conn(0x7f9e840363e0 0x7f9e840388a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ea4135a20 0x7f9ea4134ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea4135300 0x7f9ea41345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 --2- 172.21.15.79:0/510071511 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f9ea409f430 0x7f9ea41cd530 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 >> 172.21.15.79:0/510071511 conn(0x7f9ea40a25b0 msgr2=0x7f9ea41363d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:42.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 shutdown_connections 2024-04-23T16:41:42.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.274+0000 7f9eaae5c640 1 -- 172.21.15.79:0/510071511 wait complete. 2024-04-23T16:41:42.295 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-23T16:41:41.331042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163328,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067712,"statfs":{"total":579820584960,"available":579653337088,"internally_reserved":0,"allocated":1236992,"data_stored":1440242,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.909224+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:40:08.909224+0000","last_peered":"2024-04-23T16:40:08.909224+0000","last_clean":"2024-04-23T16:40:08.909224+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:40:08.909224+0000","last_undegraded":"2024-04-23T16:40:08.909224+0000","last_fullsized":"2024-04-23T16:40:08.909224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.031042+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:40:09.031042+0000","last_peered":"2024-04-23T16:40:09.031042+0000","last_clean":"2024-04-23T16:40:09.031042+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:40:09.031042+0000","last_undegraded":"2024-04-23T16:40:09.031042+0000","last_fullsized":"2024-04-23T16:40:09.031042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.908626+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:40:08.908626+0000","last_peered":"2024-04-23T16:40:08.908626+0000","last_clean":"2024-04-23T16:40:08.908626+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:40:08.908626+0000","last_undegraded":"2024-04-23T16:40:08.908626+0000","last_fullsized":"2024-04-23T16:40:08.908626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.006900+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:40:09.006900+0000","last_peered":"2024-04-23T16:40:09.006900+0000","last_clean":"2024-04-23T16:40:09.006900+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:40:09.006900+0000","last_undegraded":"2024-04-23T16:40:09.006900+0000","last_fullsized":"2024-04-23T16:40:09.006900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.039969+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:40:09.039969+0000","last_peered":"2024-04-23T16:40:09.039969+0000","last_clean":"2024-04-23T16:40:09.039969+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:40:09.039969+0000","last_undegraded":"2024-04-23T16:40:09.039969+0000","last_fullsized":"2024-04-23T16:40:09.039969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.905773+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:40:08.905773+0000","last_peered":"2024-04-23T16:40:08.905773+0000","last_clean":"2024-04-23T16:40:08.905773+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:40:08.905773+0000","last_undegraded":"2024-04-23T16:40:08.905773+0000","last_fullsized":"2024-04-23T16:40:08.905773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.032667+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:40:09.032667+0000","last_peered":"2024-04-23T16:40:09.032667+0000","last_clean":"2024-04-23T16:40:09.032667+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:40:09.032667+0000","last_undegraded":"2024-04-23T16:40:09.032667+0000","last_fullsized":"2024-04-23T16:40:09.032667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.906573+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:40:08.906573+0000","last_peered":"2024-04-23T16:40:08.906573+0000","last_clean":"2024-04-23T16:40:08.906573+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:40:08.906573+0000","last_undegraded":"2024-04-23T16:40:08.906573+0000","last_fullsized":"2024-04-23T16:40:08.906573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":175,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.907848+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:40:08.907848+0000","last_peered":"2024-04-23T16:40:08.907848+0000","last_clean":"2024-04-23T16:40:08.907848+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:40:08.907848+0000","last_undegraded":"2024-04-23T16:40:08.907848+0000","last_fullsized":"2024-04-23T16:40:08.907848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607579,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:41:42.296 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T16:41:42.296 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:41:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.394+0000 7fde113e3640 10 monclient: get_monmap_and_config 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.394+0000 7fde113e3640 10 monclient: build_initial_monmap 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.394+0000 7fde113e3640 10 monclient: monmap: 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:42.399638+0000 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:42.399638+0000 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.394+0000 7fde113e3640 1 Processor -- start 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.394+0000 7fde113e3640 1 -- start start 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.394+0000 7fde113e3640 10 monclient: init 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fde0c06ca30 0x7fde0c0a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient(hunting): picked mon.noname-a con 0x7fde0c06ca30 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde0c0a32a0 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde0c0a36d0 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a17b0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c1383c0 con 0x7fde0c0a32a0 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fde0c06ce10 con 0x7fde0c06ca30 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde113e3640 10 monclient(hunting): authenticate will time out at 1056.588220s 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fde0c06ca30 0x7fde0c0a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0b7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fde0c0a36d0 0x7fde0c137cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33422/0 (socket says 172.21.15.79:33422) 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0b7fe640 1 -- 172.21.15.79:0/423156340 learned_addr learned my addr 172.21.15.79:0/423156340 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): get_auth_request con 0x7fde0c06ca30 auth_method 0 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient(hunting): handle_auth_done global_id 14142 payload 274 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 1 -- 172.21.15.79:0/423156340 >> v1:172.21.15.204:6789/0 conn(0x7fde0c0a32a0 legacy=0x7fde0c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 1 -- 172.21.15.79:0/423156340 >> v1:172.21.15.79:6789/0 conn(0x7fde0c0a36d0 legacy=0x7fde0c137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient: _finish_hunting 0 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 1 monclient: found mon.noname-a 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.398+0000 7fde0affd640 1 -- 172.21.15.79:0/423156340 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c0a14d0 con 0x7fde0c06ca30 2024-04-23T16:41:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0affd640 1 -- 172.21.15.79:0/423156340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c06ca30 msgr2=0x7fde0c0a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0affd640 1 --2- 172.21.15.79:0/423156340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c06ca30 0x7fde0c0a4c80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0097c0 tx=0x7fddfc01c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28901bfcf7fb3b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 1 -- 172.21.15.79:0/423156340 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0b0070 con 0x7fde0c06ca30 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: dump: 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: _finish_auth 0 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: _check_auth_tickets 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 1 -- 172.21.15.79:0/423156340 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddfc0ab600 con 0x7fde0c06ca30 2024-04-23T16:41:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 5 monclient: authenticate success, global_id 14142 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 1 -- 172.21.15.79:0/423156340 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0ab9d0 con 0x7fde0c06ca30 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 4 set_mon_vals no callback set 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0a7fc640 10 monclient: dump: 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: shutdown 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- 172.21.15.79:0/423156340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c06ca30 msgr2=0x7fde0c0a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 --2- 172.21.15.79:0/423156340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c06ca30 0x7fde0c0a4c80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0097c0 tx=0x7fddfc01c100 comp rx=0 tx=0).stop 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- 172.21.15.79:0/423156340 shutdown_connections 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 --2- 172.21.15.79:0/423156340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c06ca30 0x7fde0c0a4c80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- 172.21.15.79:0/423156340 >> 172.21.15.79:0/423156340 conn(0x7fde0c0aa370 msgr2=0x7fde0c0a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- 172.21.15.79:0/423156340 shutdown_connections 2024-04-23T16:41:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- 172.21.15.79:0/423156340 wait complete. 2024-04-23T16:41:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: build_initial_monmap 2024-04-23T16:41:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: monmap: 2024-04-23T16:41:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 Processor -- start 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- start start 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: init 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:41:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde0c06ca30 0x7fde0c130b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): picked mon.noname-b con 0x7fde0c06ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fde0c0a32a0 0x7fde0c1310b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): picked mon.noname-c con 0x7fde0c0a32a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 0x7fde0c1354c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): picked mon.noname-a con 0x7fde0c0a36d0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1436a0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c143520 con 0x7fde0c06ca30 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fde0c143820 con 0x7fde0c0a32a0 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde113e3640 10 monclient(hunting): authenticate will time out at 1056.595700s 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 0x7fde0c1354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fde0c0a32a0 0x7fde0c1310b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 0x7fde0c1354c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53694/0 (socket says 172.21.15.79:53694) 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 1 -- 172.21.15.79:0/3869125245 learned_addr learned my addr 172.21.15.79:0/3869125245 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): get_auth_request con 0x7fde0c0a36d0 auth_method 0 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): get_auth_request con 0x7fde0c0a32a0 auth_method 0 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.402+0000 7fde03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0affd640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde0c06ca30 0x7fde0c130b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0affd640 10 monclient(hunting): get_auth_request con 0x7fde0c06ca30 auth_method 0 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0affd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 10 monclient(hunting): handle_auth_done global_id 14145 payload 824 2024-04-23T16:41:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 -- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fde0c0a32a0 msgr2=0x7fde0c1310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fde0c0a32a0 0x7fde0c1310b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 -- 172.21.15.79:0/3869125245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde0c06ca30 msgr2=0x7fde0c130b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde0c06ca30 0x7fde0c130b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 monclient: found mon.noname-a 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 -- 172.21.15.79:0/3869125245 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c1385b0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde03fff640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fde0c0a32a0 0x7fde0c1310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 -- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 msgr2=0x7fde0c1354c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde0b7fe640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 0x7fde0c1354c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde0400caf0 tx=0x7fde0400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04012400 con 0x7fde0c0a36d0 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: dump: 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde040125c0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040129b0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 10 monclient: dump: 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde09ffb640 4 set_mon_vals no callback set 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 5 monclient: authenticate success, global_id 14145 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 monclient: _renew_subs 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c0a4060 con 0x7fde0c0a36d0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 monclient: _renew_subs 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde0c1370d0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fde04012b90 con 0x7fde0c0a36d0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0c0692e0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.406+0000 7fde037fe640 1 --2- 172.21.15.79:0/3869125245 >> v2:172.21.15.204:6812/722020384 conn(0x7fddec0363e0 0x7fddec0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.410+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fde04049600 con 0x7fde0c0a36d0 2024-04-23T16:41:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.410+0000 7fde0affd640 1 --2- 172.21.15.79:0/3869125245 >> v2:172.21.15.204:6812/722020384 conn(0x7fddec0363e0 0x7fddec0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.410+0000 7fde0affd640 10 monclient: get_auth_request con 0x7fddec0363e0 auth_method 0 2024-04-23T16:41:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.410+0000 7fde0affd640 1 --2- 172.21.15.79:0/3869125245 >> v2:172.21.15.204:6812/722020384 conn(0x7fddec0363e0 0x7fddec0388a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0ab000 tx=0x7fddfc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.414+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fde040484f0 con 0x7fde0c0a36d0 2024-04-23T16:41:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.414+0000 7fde037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.414+0000 7fde037fe640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:42.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.570+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fde0c139d60 con 0x7fddec0363e0 2024-04-23T16:41:42.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.570+0000 7fde037fe640 1 -- 172.21.15.79:0/3869125245 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33808 (secure 0 0 0) 0x7fde0c139d60 con 0x7fddec0363e0 2024-04-23T16:41:42.577 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:41:42.577 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:41:42.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 >> v2:172.21.15.204:6812/722020384 conn(0x7fddec0363e0 msgr2=0x7fddec0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 --2- 172.21.15.79:0/3869125245 >> v2:172.21.15.204:6812/722020384 conn(0x7fddec0363e0 0x7fddec0388a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0ab000 tx=0x7fddfc008040 comp rx=0 tx=0).stop 2024-04-23T16:41:42.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 10 monclient: shutdown 2024-04-23T16:41:42.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:42.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 msgr2=0x7fde0c1354c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 0x7fde0c1354c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde0400caf0 tx=0x7fde0400a060 comp rx=0 tx=0).stop 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 shutdown_connections 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 --2- 172.21.15.79:0/3869125245 >> v2:172.21.15.204:6812/722020384 conn(0x7fddec0363e0 0x7fddec0388a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde0c0a36d0 0x7fde0c1354c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fde0c0a32a0 0x7fde0c1310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 --2- 172.21.15.79:0/3869125245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde0c06ca30 0x7fde0c130b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 >> 172.21.15.79:0/3869125245 conn(0x7fde0c0aa370 msgr2=0x7fde0c0a7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 shutdown_connections 2024-04-23T16:41:42.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.574+0000 7fde113e3640 1 -- 172.21.15.79:0/3869125245 wait complete. 2024-04-23T16:41:42.595 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-23T16:41:41.331042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163328,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067712,"statfs":{"total":579820584960,"available":579653337088,"internally_reserved":0,"allocated":1236992,"data_stored":1440242,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.909224+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:40:08.909224+0000","last_peered":"2024-04-23T16:40:08.909224+0000","last_clean":"2024-04-23T16:40:08.909224+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:40:08.909224+0000","last_undegraded":"2024-04-23T16:40:08.909224+0000","last_fullsized":"2024-04-23T16:40:08.909224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.031042+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:40:09.031042+0000","last_peered":"2024-04-23T16:40:09.031042+0000","last_clean":"2024-04-23T16:40:09.031042+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:40:09.031042+0000","last_undegraded":"2024-04-23T16:40:09.031042+0000","last_fullsized":"2024-04-23T16:40:09.031042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.908626+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:40:08.908626+0000","last_peered":"2024-04-23T16:40:08.908626+0000","last_clean":"2024-04-23T16:40:08.908626+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:40:08.908626+0000","last_undegraded":"2024-04-23T16:40:08.908626+0000","last_fullsized":"2024-04-23T16:40:08.908626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.006900+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:40:09.006900+0000","last_peered":"2024-04-23T16:40:09.006900+0000","last_clean":"2024-04-23T16:40:09.006900+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:40:09.006900+0000","last_undegraded":"2024-04-23T16:40:09.006900+0000","last_fullsized":"2024-04-23T16:40:09.006900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.039969+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:40:09.039969+0000","last_peered":"2024-04-23T16:40:09.039969+0000","last_clean":"2024-04-23T16:40:09.039969+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:40:09.039969+0000","last_undegraded":"2024-04-23T16:40:09.039969+0000","last_fullsized":"2024-04-23T16:40:09.039969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.905773+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:40:08.905773+0000","last_peered":"2024-04-23T16:40:08.905773+0000","last_clean":"2024-04-23T16:40:08.905773+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:40:08.905773+0000","last_undegraded":"2024-04-23T16:40:08.905773+0000","last_fullsized":"2024-04-23T16:40:08.905773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.032667+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:40:09.032667+0000","last_peered":"2024-04-23T16:40:09.032667+0000","last_clean":"2024-04-23T16:40:09.032667+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:40:09.032667+0000","last_undegraded":"2024-04-23T16:40:09.032667+0000","last_fullsized":"2024-04-23T16:40:09.032667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.906573+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:40:08.906573+0000","last_peered":"2024-04-23T16:40:08.906573+0000","last_clean":"2024-04-23T16:40:08.906573+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:40:08.906573+0000","last_undegraded":"2024-04-23T16:40:08.906573+0000","last_fullsized":"2024-04-23T16:40:08.906573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":175,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.907848+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:40:08.907848+0000","last_peered":"2024-04-23T16:40:08.907848+0000","last_clean":"2024-04-23T16:40:08.907848+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:40:08.907848+0000","last_undegraded":"2024-04-23T16:40:08.907848+0000","last_fullsized":"2024-04-23T16:40:08.907848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607579,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:41:42.596 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T16:41:42.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.710+0000 7f11f6276640 10 monclient: get_monmap_and_config 2024-04-23T16:41:42.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.710+0000 7f11f6276640 10 monclient: build_initial_monmap 2024-04-23T16:41:42.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.710+0000 7f11f6276640 10 monclient: monmap: 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:42.716434+0000 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:42.716434+0000 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:42.716 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.710+0000 7f11f6276640 1 Processor -- start 2024-04-23T16:41:42.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.710+0000 7f11f6276640 1 -- start start 2024-04-23T16:41:42.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.710+0000 7f11f6276640 10 monclient: init 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f11f0135a20 0x7f11f0135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): picked mon.noname-a con 0x7f11f0135a20 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11f009f430 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f11f0135300 0x7f11f009c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): picked mon.noname-c con 0x7f11f0135300 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11f0072f80 con 0x7f11f009f430 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f11f009fb10 con 0x7f11f0135a20 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f11f009fdf0 con 0x7f11f0135300 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:42.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f6276640 10 monclient(hunting): authenticate will time out at 1056.904311s 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f11f0135a20 0x7f11f0135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11ef7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f11f009f430 0x7f11f009ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33426/0 (socket says 172.21.15.79:33426) 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11ef7fe640 1 -- 172.21.15.79:0/75805154 learned_addr learned my addr 172.21.15.79:0/75805154 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): get_auth_request con 0x7f11f0135a20 auth_method 0 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11f4813640 1 --2- 172.21.15.79:0/75805154 >> v2:172.21.15.204:3300/0 conn(0x7f11f0135300 0x7f11f009c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient(hunting): handle_auth_done global_id 14148 payload 274 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 1 -- 172.21.15.79:0/75805154 >> v2:172.21.15.204:3300/0 conn(0x7f11f0135300 msgr2=0x7f11f009c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 1 --2- 172.21.15.79:0/75805154 >> v2:172.21.15.204:3300/0 conn(0x7f11f0135300 0x7f11f009c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 1 -- 172.21.15.79:0/75805154 >> v1:172.21.15.79:6789/0 conn(0x7f11f009f430 legacy=0x7f11f009ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:42.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:42.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 1 monclient: found mon.noname-a 2024-04-23T16:41:42.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:42.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.714+0000 7f11effff640 1 -- 172.21.15.79:0/75805154 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f006de30 con 0x7f11f0135a20 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 1 -- 172.21.15.79:0/75805154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135a20 msgr2=0x7f11f0135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 1 --2- 172.21.15.79:0/75805154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135a20 0x7f11f0135e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11f0136400 tx=0x7f11e001c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62e013c53f16b1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 1 -- 172.21.15.79:0/75805154 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00b0070 con 0x7f11f0135a20 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: dump: 2024-04-23T16:41:42.723 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: _finish_auth 0 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:42.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 1 -- 172.21.15.79:0/75805154 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e00ab600 con 0x7f11f0135a20 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 1 -- 172.21.15.79:0/75805154 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00ab9d0 con 0x7f11f0135a20 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11eeffd640 10 monclient: dump: 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 5 monclient: authenticate success, global_id 14148 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 4 set_mon_vals no callback set 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: shutdown 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- 172.21.15.79:0/75805154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135a20 msgr2=0x7f11f0135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 --2- 172.21.15.79:0/75805154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135a20 0x7f11f0135e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11f0136400 tx=0x7f11e001c300 comp rx=0 tx=0).stop 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- 172.21.15.79:0/75805154 shutdown_connections 2024-04-23T16:41:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 --2- 172.21.15.79:0/75805154 >> v2:172.21.15.204:3300/0 conn(0x7f11f0135300 0x7f11f009c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 --2- 172.21.15.79:0/75805154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135a20 0x7f11f0135e00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- 172.21.15.79:0/75805154 >> 172.21.15.79:0/75805154 conn(0x7f11f00a25b0 msgr2=0x7f11f00a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- 172.21.15.79:0/75805154 shutdown_connections 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- 172.21.15.79:0/75805154 wait complete. 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: build_initial_monmap 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: monmap: 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:42.728 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 Processor -- start 2024-04-23T16:41:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- start start 2024-04-23T16:41:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: init 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f11f009f430 0x7f11f013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): picked mon.noname-c con 0x7f11f009f430 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 0x7f11f0134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): picked mon.noname-a con 0x7f11f0135300 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11f0135a20 0x7f11f0134840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): picked mon.noname-b con 0x7f11f0135a20 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01400b0 con 0x7f11f0135300 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f11f009c860 con 0x7f11f0135a20 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f11f0140230 con 0x7f11f009f430 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f6276640 10 monclient(hunting): authenticate will time out at 1056.911340s 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 0x7f11f0134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f11f009f430 0x7f11f013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 0x7f11f0134300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53708/0 (socket says 172.21.15.79:53708) 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): get_auth_request con 0x7f11f009f430 auth_method 0 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 -- 172.21.15.79:0/550101323 learned_addr learned my addr 172.21.15.79:0/550101323 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): get_auth_request con 0x7f11f0135300 auth_method 0 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient(hunting): handle_auth_done global_id 14151 payload 824 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11f4813640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11f0135a20 0x7f11f0134840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 -- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f11f009f430 msgr2=0x7f11f013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f11f009f430 0x7f11f013c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 -- 172.21.15.79:0/550101323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11f0135a20 msgr2=0x7f11f0134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11f0135a20 0x7f11f0134840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 monclient: found mon.noname-a 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11ef7fe640 1 -- 172.21.15.79:0/550101323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f013cc60 con 0x7f11f0135300 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.718+0000 7f11effff640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f11f009f430 0x7f11f013c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ef7fe640 1 -- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 msgr2=0x7f11f0134300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ef7fe640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 0x7f11f0134300 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11e800cc10 tx=0x7f11e800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8012400 con 0x7f11f0135300 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: dump: 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.737 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: _finish_auth 0 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: _check_auth_tickets 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e80125e0 con 0x7f11f0135300 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e80129d0 con 0x7f11f0135300 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 5 monclient: authenticate success, global_id 14151 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 10 monclient: dump: 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:42.738 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11edffb640 4 set_mon_vals no callback set 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 monclient: _renew_subs 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f0072c70 con 0x7f11f0135300 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 monclient: _renew_subs 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f009dd20 con 0x7f11f0135300 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f11e8012bb0 con 0x7f11f0135300 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f00692e0 con 0x7f11f0135300 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 1 --2- 172.21.15.79:0/550101323 >> v2:172.21.15.204:6812/722020384 conn(0x7f11d0036430 0x7f11d00388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f11e8049650 con 0x7f11f0135300 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11effff640 1 --2- 172.21.15.79:0/550101323 >> v2:172.21.15.204:6812/722020384 conn(0x7f11d0036430 0x7f11d00388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.722+0000 7f11effff640 10 monclient: get_auth_request con 0x7f11d0036430 auth_method 0 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.726+0000 7f11effff640 1 --2- 172.21.15.79:0/550101323 >> v2:172.21.15.204:6812/722020384 conn(0x7f11d0036430 0x7f11d00388f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f11f006dbd0 tx=0x7f11e00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.730+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f11e801a360 con 0x7f11f0135300 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.730+0000 7f11ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:42.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.730+0000 7f11ecff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:42.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.886+0000 7f11f6276640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:41:42.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.886+0000 7f11f6276640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:41:42.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.886+0000 7f11f6276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:42.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.886+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11f0120650 con 0x7f11f0135300 2024-04-23T16:41:42.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11ecff9640 1 -- 172.21.15.79:0/550101323 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+10799 (secure 0 0 0) 0x7f11e8048540 con 0x7f11f0135300 2024-04-23T16:41:42.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-23T16:41:42.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11ecff9640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:41:42.893 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:41:42.894 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"fsid":"0469449d-bc82-4171-9959-1bb87fc3dec3","created":"2024-04-23T16:39:30.490752+0000","modified":"2024-04-23T16:40:08.898040+0000","last_up_change":"2024-04-23T16:39:41.690496+0000","last_in_change":"2024-04-23T16:39:33.300300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:39:41.354561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-23T16:39:42.435523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2728fa88-43d5-4a55-998c-4b58cfb56136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":987647954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":987647954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":987647954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":987647954}]},"public_addr":"172.21.15.79:6802/987647954","cluster_addr":"172.21.15.79:6805/987647954","heartbeat_back_addr":"172.21.15.79:6810/987647954","heartbeat_front_addr":"172.21.15.79:6808/987647954","state":["exists","up"]},{"osd":1,"uuid":"5dcf8600-d4ef-47a3-8e79-a9fde5ab876c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3615156451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":3615156451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":3615156451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":3615156451}]},"public_addr":"172.21.15.79:6803/3615156451","cluster_addr":"172.21.15.79:6806/3615156451","heartbeat_back_addr":"172.21.15.79:6811/3615156451","heartbeat_front_addr":"172.21.15.79:6809/3615156451","state":["exists","up"]},{"osd":2,"uuid":"d079e76b-d697-4efc-a10e-52178c04b550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3678369310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":3678369310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":3678369310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":3678369310}]},"public_addr":"172.21.15.79:6800/3678369310","cluster_addr":"172.21.15.79:6801/3678369310","heartbeat_back_addr":"172.21.15.79:6807/3678369310","heartbeat_front_addr":"172.21.15.79:6804/3678369310","state":["exists","up"]},{"osd":3,"uuid":"fa572109-89d7-4f3d-bd5a-312932eb60b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3194378740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":3194378740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":3194378740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3194378740}]},"public_addr":"172.21.15.204:6800/3194378740","cluster_addr":"172.21.15.204:6801/3194378740","heartbeat_back_addr":"172.21.15.204:6803/3194378740","heartbeat_front_addr":"172.21.15.204:6802/3194378740","state":["exists","up"]},{"osd":4,"uuid":"b34e743a-ccc5-46b6-8512-69cd9d149b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3419064638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":3419064638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":3419064638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3419064638}]},"public_addr":"172.21.15.204:6804/3419064638","cluster_addr":"172.21.15.204:6805/3419064638","heartbeat_back_addr":"172.21.15.204:6807/3419064638","heartbeat_front_addr":"172.21.15.204:6806/3419064638","state":["exists","up"]},{"osd":5,"uuid":"8c45a63c-15d6-4ee9-a8dd-13edd0f0d7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":3961528928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":3961528928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":3961528928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":3961528928}]},"public_addr":"172.21.15.204:6808/3961528928","cluster_addr":"172.21.15.204:6809/3961528928","heartbeat_back_addr":"172.21.15.204:6811/3961528928","heartbeat_front_addr":"172.21.15.204:6810/3961528928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.690894+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.747794+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.664067+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.480934+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.716414+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:39:39.584524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:41:42.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 >> v2:172.21.15.204:6812/722020384 conn(0x7f11d0036430 msgr2=0x7f11d00388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 --2- 172.21.15.79:0/550101323 >> v2:172.21.15.204:6812/722020384 conn(0x7f11d0036430 0x7f11d00388f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f11f006dbd0 tx=0x7f11e00a6000 comp rx=0 tx=0).stop 2024-04-23T16:41:42.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 10 monclient: shutdown 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 msgr2=0x7f11f0134300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 0x7f11f0134300 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11e800cc10 tx=0x7f11e800a430 comp rx=0 tx=0).stop 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 shutdown_connections 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 --2- 172.21.15.79:0/550101323 >> v2:172.21.15.204:6812/722020384 conn(0x7f11d0036430 0x7f11d00388f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11f0135a20 0x7f11f0134840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11f0135300 0x7f11f0134300 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 --2- 172.21.15.79:0/550101323 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f11f009f430 0x7f11f013c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 >> 172.21.15.79:0/550101323 conn(0x7f11f00a25b0 msgr2=0x7f11f00a39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 shutdown_connections 2024-04-23T16:41:42.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:42.890+0000 7f11f6276640 1 -- 172.21.15.79:0/550101323 wait complete. 2024-04-23T16:41:43.911 INFO:tasks.ceph:Scrubbing osd.0 2024-04-23T16:41:43.912 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-23T16:41:44.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient: get_monmap_and_config 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient: monmap: 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:44.046536+0000 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:44.046536+0000 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:44.046 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:44.047 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 Processor -- start 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 -- start start 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient: init 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77300a14c0 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f77300a18a0 0x7f77300a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f77300a18a0 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f773013bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f7730139740 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773009e9f0 con 0x7f77300a14c0 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f773009f040 con 0x7f77300a18a0 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f773009e710 con 0x7f7730139740 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7735e2a640 10 monclient(hunting): authenticate will time out at 1058.234907s 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772f7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f77300a14c0 0x7f773009f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33438/0 (socket says 172.21.15.79:33438) 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f773013bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772f7fe640 1 -- 172.21.15.79:0/2150988170 learned_addr learned my addr 172.21.15.79:0/2150988170 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): get_auth_request con 0x7f7730139740 auth_method 0 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7727fff640 1 --2- 172.21.15.79:0/2150988170 >> v2:172.21.15.204:3300/0 conn(0x7f77300a18a0 0x7f77300a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7727fff640 10 monclient(hunting): get_auth_request con 0x7f77300a18a0 auth_method 0 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7727fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f7727fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient(hunting): handle_auth_done global_id 4430 payload 274 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 1 -- 172.21.15.79:0/2150988170 >> v2:172.21.15.204:3300/0 conn(0x7f77300a18a0 msgr2=0x7f77300a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 1 --2- 172.21.15.79:0/2150988170 >> v2:172.21.15.204:3300/0 conn(0x7f77300a18a0 0x7f77300a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 1 -- 172.21.15.79:0/2150988170 >> v1:172.21.15.79:6789/0 conn(0x7f77300a14c0 legacy=0x7f773009f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 1 monclient: found mon.noname-b 2024-04-23T16:41:44.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.042+0000 7f772ffff640 1 -- 172.21.15.79:0/2150988170 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773009ecd0 con 0x7f7730139740 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 -- 172.21.15.79:0/2150988170 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 msgr2=0x7f773013bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 --2- 172.21.15.79:0/2150988170 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f773013bb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7728009d90 tx=0x7f772801d280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e51738319a7b4fef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 1 -- 172.21.15.79:0/2150988170 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280b2070 con 0x7f7730139740 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: dump: 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.054 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 1 -- 172.21.15.79:0/2150988170 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280a58c0 con 0x7f7730139740 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 5 monclient: authenticate success, global_id 4430 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:44.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 4 set_mon_vals no callback set 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 1 -- 172.21.15.79:0/2150988170 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280a5c90 con 0x7f7730139740 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772effd640 10 monclient: dump: 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.056 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: shutdown 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- 172.21.15.79:0/2150988170 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 msgr2=0x7f773013bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 --2- 172.21.15.79:0/2150988170 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f773013bb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7728009d90 tx=0x7f772801d280 comp rx=0 tx=0).stop 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- 172.21.15.79:0/2150988170 shutdown_connections 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 --2- 172.21.15.79:0/2150988170 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f773013bb30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 --2- 172.21.15.79:0/2150988170 >> v2:172.21.15.204:3300/0 conn(0x7f77300a18a0 0x7f77300a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- 172.21.15.79:0/2150988170 >> 172.21.15.79:0/2150988170 conn(0x7f77300a25b0 msgr2=0x7f77300a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- 172.21.15.79:0/2150988170 shutdown_connections 2024-04-23T16:41:44.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- 172.21.15.79:0/2150988170 wait complete. 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: monmap: 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:44.062 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 Processor -- start 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- start start 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: init 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f77300a14c0 0x7f7730070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f77300a14c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77300a18a0 0x7f773011c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f77300a18a0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f77301332d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f7730139740 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f773013ea90 con 0x7f77300a18a0 2024-04-23T16:41:44.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f773013e910 con 0x7f77300a14c0 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f773013ec10 con 0x7f7730139740 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7735e2a640 10 monclient(hunting): authenticate will time out at 1058.242948s 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77300a18a0 0x7f773011c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f77301332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f77301332d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45536/0 (socket says 172.21.15.79:45536) 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 -- 172.21.15.79:0/2069613623 learned_addr learned my addr 172.21.15.79:0/2069613623 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): get_auth_request con 0x7f77300a18a0 auth_method 0 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): get_auth_request con 0x7f7730139740 auth_method 0 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772f7fe640 1 --2- 172.21.15.79:0/2069613623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f77300a14c0 0x7f7730070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772f7fe640 10 monclient(hunting): get_auth_request con 0x7f77300a14c0 auth_method 0 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient(hunting): handle_auth_done global_id 4433 payload 824 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 -- 172.21.15.79:0/2069613623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f77300a14c0 msgr2=0x7f7730070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 --2- 172.21.15.79:0/2069613623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f77300a14c0 0x7f7730070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 -- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77300a18a0 msgr2=0x7f773011c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 --2- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77300a18a0 0x7f773011c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 monclient: found mon.noname-c 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f772ffff640 1 -- 172.21.15.79:0/2069613623 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7730134150 con 0x7f7730139740 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.050+0000 7f7727fff640 1 --2- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77300a18a0 0x7f773011c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:44.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772ffff640 1 -- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 msgr2=0x7f77301332d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772ffff640 1 --2- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f77301332d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77280b0040 tx=0x7f7728019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280bc070 con 0x7f7730139740 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: dump: 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.067 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280ad850 con 0x7f7730139740 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 5 monclient: authenticate success, global_id 4433 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 10 monclient: _renew_subs 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773011e430 con 0x7f7730139740 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280b2040 con 0x7f7730139740 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: dump: 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 10 monclient: _renew_subs 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772e7fc640 4 set_mon_vals no callback set 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f773011cfe0 con 0x7f7730139740 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f773011e7d0 con 0x7f7730139740 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f77280b5070 con 0x7f7730139740 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.204:6812/722020384 conn(0x7f7710036410 0x7f77100388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f77280e37b0 con 0x7f7730139740 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.79:6802/987647954 conn(0x7f771003bf20 0x7f771003e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 --> v2:172.21.15.79:6802/987647954 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f771003e9b0 con 0x7f771003bf20 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f77280ea050 con 0x7f7730139740 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772cff9640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7727fff640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.79:6802/987647954 conn(0x7f771003bf20 0x7f771003e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f7727fff640 10 monclient: get_auth_request con 0x7f771003bf20 auth_method 0 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772f7fe640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.204:6812/722020384 conn(0x7f7710036410 0x7f77100388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.054+0000 7f772f7fe640 10 monclient: get_auth_request con 0x7f7710036410 auth_method 0 2024-04-23T16:41:44.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.058+0000 7f772f7fe640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.204:6812/722020384 conn(0x7f7710036410 0x7f77100388d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7720007fd0 tx=0x7f7720009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.058+0000 7f7727fff640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.79:6802/987647954 conn(0x7f771003bf20 0x7f771003e340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.062+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== osd.0 v2:172.21.15.79:6802/987647954 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f771003e9b0 con 0x7f771003bf20 2024-04-23T16:41:44.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 --> v2:172.21.15.79:6802/987647954 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7730133a60 con 0x7f771003bf20 2024-04-23T16:41:44.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f772cff9640 1 -- 172.21.15.79:0/2069613623 <== osd.0 v2:172.21.15.79:6802/987647954 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7730133a60 con 0x7f771003bf20 2024-04-23T16:41:44.078 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T16:41:44.079 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 >> v2:172.21.15.79:6802/987647954 conn(0x7f771003bf20 msgr2=0x7f771003e340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.79:6802/987647954 conn(0x7f771003bf20 0x7f771003e340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 >> v2:172.21.15.204:6812/722020384 conn(0x7f7710036410 msgr2=0x7f77100388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 --2- 172.21.15.79:0/2069613623 >> v2:172.21.15.204:6812/722020384 conn(0x7f7710036410 0x7f77100388d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7720007fd0 tx=0x7f7720009040 comp rx=0 tx=0).stop 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 10 monclient: shutdown 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 msgr2=0x7f77301332d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 --2- 172.21.15.79:0/2069613623 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7730139740 0x7f77301332d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77280b0040 tx=0x7f7728019620 comp rx=0 tx=0).stop 2024-04-23T16:41:44.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f772ffff640 1 -- 172.21.15.79:0/2069613623 reap_dead start 2024-04-23T16:41:44.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 shutdown_connections 2024-04-23T16:41:44.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 >> 172.21.15.79:0/2069613623 conn(0x7f77300a25b0 msgr2=0x7f773009ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 shutdown_connections 2024-04-23T16:41:44.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.074+0000 7f7735e2a640 1 -- 172.21.15.79:0/2069613623 wait complete. 2024-04-23T16:41:44.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient: get_monmap_and_config 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient: monmap: 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:44.212005+0000 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:44.212005+0000 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:44.211 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:44.212 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:44.212 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:44.212 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:44.212 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 1 Processor -- start 2024-04-23T16:41:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 1 -- start start 2024-04-23T16:41:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient: init 2024-04-23T16:41:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-23T16:41:44.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbedc06cab0 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbedc0a3c30 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fbedc137b80 0x7fbedc1400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient(hunting): picked mon.noname-a con 0x7fbedc137b80 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbedc138590 con 0x7fbedc0a3c30 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbedc0a5750 con 0x7fbedc06cab0 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fbedc06ce90 con 0x7fbedc137b80 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.206+0000 7fbee3543640 10 monclient(hunting): authenticate will time out at 1058.399868s 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fbedc137b80 0x7fbedc1400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee0ade640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fbedc0a3c30 0x7fbedc13c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33448/0 (socket says 172.21.15.79:33448) 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee0ade640 1 -- 172.21.15.79:0/1783150670 learned_addr learned my addr 172.21.15.79:0/1783150670 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): get_auth_request con 0x7fbedc137b80 auth_method 0 2024-04-23T16:41:44.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient(hunting): handle_auth_done global_id 14157 payload 274 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 1 -- 172.21.15.79:0/1783150670 >> v1:172.21.15.204:6789/0 conn(0x7fbedc06cab0 legacy=0x7fbedc0a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 1 -- 172.21.15.79:0/1783150670 >> v1:172.21.15.79:6789/0 conn(0x7fbedc0a3c30 legacy=0x7fbedc13c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 1 monclient: found mon.noname-a 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:44.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.210+0000 7fbee1ae0640 1 -- 172.21.15.79:0/1783150670 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbedc0a17d0 con 0x7fbedc137b80 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee1ae0640 1 -- 172.21.15.79:0/1783150670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc137b80 msgr2=0x7fbedc1400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee1ae0640 1 --2- 172.21.15.79:0/1783150670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc137b80 0x7fbedc1400f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0097c0 tx=0x7fbecc01c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81ddf5e61f3ad0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 1 -- 172.21.15.79:0/1783150670 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0b0070 con 0x7fbedc137b80 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: dump: 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 1 -- 172.21.15.79:0/1783150670 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbecc0ab600 con 0x7fbedc137b80 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 5 monclient: authenticate success, global_id 14157 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 1 -- 172.21.15.79:0/1783150670 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0ab9d0 con 0x7fbedc137b80 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 4 set_mon_vals no callback set 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbed3fff640 10 monclient: dump: 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.221 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: shutdown 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- 172.21.15.79:0/1783150670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc137b80 msgr2=0x7fbedc1400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 --2- 172.21.15.79:0/1783150670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc137b80 0x7fbedc1400f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0097c0 tx=0x7fbecc01c2e0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- 172.21.15.79:0/1783150670 shutdown_connections 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 --2- 172.21.15.79:0/1783150670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc137b80 0x7fbedc1400f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- 172.21.15.79:0/1783150670 >> 172.21.15.79:0/1783150670 conn(0x7fbedc0aa3d0 msgr2=0x7fbedc0a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- 172.21.15.79:0/1783150670 shutdown_connections 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- 172.21.15.79:0/1783150670 wait complete. 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: monmap: 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:44.223 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:44.224 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 Processor -- start 2024-04-23T16:41:44.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- start start 2024-04-23T16:41:44.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: init 2024-04-23T16:41:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 0x7fbedc1359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): picked mon.noname-c con 0x7fbedc06cab0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 0x7fbedc135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): picked mon.noname-a con 0x7fbedc0a3c30 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbedc137b80 0x7fbedc1cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): picked mon.noname-b con 0x7fbedc137b80 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc143240 con 0x7fbedc0a3c30 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc1430c0 con 0x7fbedc137b80 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbedc1433c0 con 0x7fbedc06cab0 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee3543640 10 monclient(hunting): authenticate will time out at 1058.406980s 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 0x7fbedc135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 0x7fbedc1359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 0x7fbedc135f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53734/0 (socket says 172.21.15.79:53734) 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 1 -- 172.21.15.79:0/4185799663 learned_addr learned my addr 172.21.15.79:0/4185799663 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): get_auth_request con 0x7fbedc0a3c30 auth_method 0 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): get_auth_request con 0x7fbedc06cab0 auth_method 0 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 -- 172.21.15.79:0/4185799663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbedc137b80 msgr2=0x7fbedc1cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbedc137b80 0x7fbedc1cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 -- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 msgr2=0x7fbedc135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 0x7fbedc135f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 monclient: found mon.noname-c 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee12df640 1 -- 172.21.15.79:0/4185799663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbedc0a4080 con 0x7fbedc06cab0 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.214+0000 7fbee0ade640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 0x7fbedc135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee12df640 1 -- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 msgr2=0x7fbedc1359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee12df640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 0x7fbedc1359f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbed400c390 tx=0x7fbed400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed40119a0 con 0x7fbedc06cab0 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: dump: 2024-04-23T16:41:44.231 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed4017070 con 0x7fbedc06cab0 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 5 monclient: authenticate success, global_id 4436 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 monclient: _renew_subs 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbedc13fa90 con 0x7fbedc06cab0 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4011d80 con 0x7fbedc06cab0 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 10 monclient: dump: 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.233 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed37fe640 4 set_mon_vals no callback set 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 monclient: _renew_subs 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbedc13e710 con 0x7fbedc06cab0 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:44.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbedc0692e0 con 0x7fbedc06cab0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fbed401c460 con 0x7fbedc06cab0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 1 --2- 172.21.15.79:0/4185799663 >> v2:172.21.15.204:6812/722020384 conn(0x7fbec80363e0 0x7fbec80388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fbed4048cf0 con 0x7fbedc06cab0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee0ade640 1 --2- 172.21.15.79:0/4185799663 >> v2:172.21.15.204:6812/722020384 conn(0x7fbec80363e0 0x7fbec80388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.218+0000 7fbee0ade640 10 monclient: get_auth_request con 0x7fbec80363e0 auth_method 0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.222+0000 7fbee0ade640 1 --2- 172.21.15.79:0/4185799663 >> v2:172.21.15.204:6812/722020384 conn(0x7fbec80363e0 0x7fbec80388a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbedc13cf10 tx=0x7fbed8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.222+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbed4012420 con 0x7fbedc06cab0 2024-04-23T16:41:44.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.222+0000 7fbed1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:44.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.222+0000 7fbed1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:44.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.374+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fbedc072040 con 0x7fbec80363e0 2024-04-23T16:41:44.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.374+0000 7fbed1ffb640 1 -- 172.21.15.79:0/4185799663 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbedc072040 con 0x7fbec80363e0 2024-04-23T16:41:44.379 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 0 to deep-scrub 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 >> v2:172.21.15.204:6812/722020384 conn(0x7fbec80363e0 msgr2=0x7fbec80388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 --2- 172.21.15.79:0/4185799663 >> v2:172.21.15.204:6812/722020384 conn(0x7fbec80363e0 0x7fbec80388a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbedc13cf10 tx=0x7fbed8009040 comp rx=0 tx=0).stop 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 10 monclient: shutdown 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 msgr2=0x7fbedc1359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 0x7fbedc1359f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbed400c390 tx=0x7fbed400c850 comp rx=0 tx=0).stop 2024-04-23T16:41:44.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 shutdown_connections 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 --2- 172.21.15.79:0/4185799663 >> v2:172.21.15.204:6812/722020384 conn(0x7fbec80363e0 0x7fbec80388a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbedc137b80 0x7fbedc1cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbedc0a3c30 0x7fbedc135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 --2- 172.21.15.79:0/4185799663 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbedc06cab0 0x7fbedc1359f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 >> 172.21.15.79:0/4185799663 conn(0x7fbedc0aa3d0 msgr2=0x7fbedc0a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 shutdown_connections 2024-04-23T16:41:44.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.378+0000 7fbee3543640 1 -- 172.21.15.79:0/4185799663 wait complete. 2024-04-23T16:41:44.396 INFO:tasks.ceph:Scrubbing osd.1 2024-04-23T16:41:44.397 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-23T16:41:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient: get_monmap_and_config 2024-04-23T16:41:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient: monmap: 2024-04-23T16:41:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:44.514010+0000 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:44.514010+0000 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:44.514 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:44.515 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 1 Processor -- start 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 1 -- start start 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient: init 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f35080a14c0 0x7f350809f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient(hunting): picked mon.noname-a con 0x7f35080a14c0 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f350809f910 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35080a1fb0 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350809e9f0 con 0x7f350809f910 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350809ecd0 con 0x7f35080a1fb0 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f350809f040 con 0x7f35080a14c0 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350f0b9640 10 monclient(hunting): authenticate will time out at 1058.701718s 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f35080a14c0 0x7f350809f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f35080a14c0 0x7f350809f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53744/0 (socket says 172.21.15.79:53744) 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 -- 172.21.15.79:0/1082475502 learned_addr learned my addr 172.21.15.79:0/1082475502 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:44.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f34fffff640 1 --1- 172.21.15.79:0/1082475502 >> v1:172.21.15.79:6789/0 conn(0x7f350809f910 0x7f35080a18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33452/0 (socket says 172.21.15.79:33452) 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): get_auth_request con 0x7f35080a14c0 auth_method 0 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient(hunting): handle_auth_done global_id 14163 payload 274 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 -- 172.21.15.79:0/1082475502 >> v1:172.21.15.204:6789/0 conn(0x7f35080a1fb0 legacy=0x7f350813b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 -- 172.21.15.79:0/1082475502 >> v1:172.21.15.79:6789/0 conn(0x7f350809f910 legacy=0x7f35080a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 monclient: found mon.noname-a 2024-04-23T16:41:44.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:44.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.510+0000 7f350ce55640 1 -- 172.21.15.79:0/1082475502 --> v2:172.21.15.79:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35080a3330 con 0x7f35080a14c0 2024-04-23T16:41:44.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350ce55640 1 -- 172.21.15.79:0/1082475502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f35080a14c0 msgr2=0x7f350809f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350ce55640 1 --2- 172.21.15.79:0/1082475502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f35080a14c0 0x7f350809f3d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3504009870 tx=0x7f350401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e43adc4017d4001 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 1 -- 172.21.15.79:0/1082475502 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040b0070 con 0x7f35080a14c0 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: dump: 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.527 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 1 -- 172.21.15.79:0/1082475502 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35040ab600 con 0x7f35080a14c0 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 1 -- 172.21.15.79:0/1082475502 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040ab9d0 con 0x7f35080a14c0 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 5 monclient: authenticate success, global_id 14163 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 4 set_mon_vals no callback set 2024-04-23T16:41:44.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f34ff7fe640 10 monclient: dump: 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.529 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 10 monclient: shutdown 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 -- 172.21.15.79:0/1082475502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f35080a14c0 msgr2=0x7f350809f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 --2- 172.21.15.79:0/1082475502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f35080a14c0 0x7f350809f3d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3504009870 tx=0x7f350401c430 comp rx=0 tx=0).stop 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 -- 172.21.15.79:0/1082475502 shutdown_connections 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 --2- 172.21.15.79:0/1082475502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f35080a14c0 0x7f350809f3d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 -- 172.21.15.79:0/1082475502 >> 172.21.15.79:0/1082475502 conn(0x7f35080a25b0 msgr2=0x7f35080a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 -- 172.21.15.79:0/1082475502 shutdown_connections 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.514+0000 7f350f0b9640 1 -- 172.21.15.79:0/1082475502 wait complete. 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient: monmap: 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 Processor -- start 2024-04-23T16:41:44.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 -- start start 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient: init 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 0x7f35081cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): picked mon.noname-a con 0x7f350809f910 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f35080a14c0 0x7f3508136550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): picked mon.noname-c con 0x7f35080a14c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35080a1fb0 0x7f350811d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): picked mon.noname-b con 0x7f35080a1fb0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f350813fd40 con 0x7f350809f910 2024-04-23T16:41:44.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f350813fbc0 con 0x7f35080a1fb0 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f350813fec0 con 0x7f35080a14c0 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350f0b9640 10 monclient(hunting): authenticate will time out at 1058.709132s 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 0x7f35081cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f35080a14c0 0x7f3508136550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 0x7f35081cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53746/0 (socket says 172.21.15.79:53746) 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 -- 172.21.15.79:0/3557290425 learned_addr learned my addr 172.21.15.79:0/3557290425 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): get_auth_request con 0x7f350809f910 auth_method 0 2024-04-23T16:41:44.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): get_auth_request con 0x7f35080a14c0 auth_method 0 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient(hunting): handle_auth_done global_id 14166 payload 824 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 -- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f35080a14c0 msgr2=0x7f3508136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f35080a14c0 0x7f3508136550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 -- 172.21.15.79:0/3557290425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35080a1fb0 msgr2=0x7f350811d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350d656640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35080a1fb0 0x7f350811d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35080a1fb0 0x7f350811d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 monclient: found mon.noname-a 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:44.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 -- 172.21.15.79:0/3557290425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3508137440 con 0x7f350809f910 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350d656640 10 monclient: get_auth_request con 0x7f35080a1fb0 auth_method 0 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f34fffff640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f35080a14c0 0x7f3508136550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350d656640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35080a1fb0 0x7f350811d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 -- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 msgr2=0x7f35081cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.518+0000 7f350ce55640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 0x7f35081cd290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f350401a060 tx=0x7f35040191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040b9020 con 0x7f350809f910 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: dump: 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.536 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35040b1040 con 0x7f350809f910 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040b0050 con 0x7f350809f910 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 5 monclient: authenticate success, global_id 14166 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 10 monclient: _renew_subs 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350809e080 con 0x7f350809f910 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34feffd640 4 set_mon_vals no callback set 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: dump: 2024-04-23T16:41:44.538 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 10 monclient: _renew_subs 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3508120ee0 con 0x7f350809f910 2024-04-23T16:41:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35081ce200 con 0x7f350809f910 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f35040bc020 con 0x7f350809f910 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.204:6812/722020384 conn(0x7f34f40363e0 0x7f34f40388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f35040e3000 con 0x7f350809f910 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.79:6803/3615156451 conn(0x7f34f403bef0 0x7f34f403e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 --> v2:172.21.15.79:6803/3615156451 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3504006800 con 0x7f34f403bef0 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f35040e55d0 con 0x7f350809f910 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fd7fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350d656640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.79:6803/3615156451 conn(0x7f34f403bef0 0x7f34f403e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f350d656640 10 monclient: get_auth_request con 0x7f34f403bef0 auth_method 0 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fffff640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.204:6812/722020384 conn(0x7f34f40363e0 0x7f34f40388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.522+0000 7f34fffff640 10 monclient: get_auth_request con 0x7f34f40363e0 auth_method 0 2024-04-23T16:41:44.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.526+0000 7f34fffff640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.204:6812/722020384 conn(0x7f34f40363e0 0x7f34f40388a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f350811e830 tx=0x7f34f8007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.526+0000 7f350d656640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.79:6803/3615156451 conn(0x7f34f403bef0 0x7f34f403e310 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.526+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== osd.1 v2:172.21.15.79:6803/3615156451 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3504006800 con 0x7f34f403bef0 2024-04-23T16:41:44.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.538+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 --> v2:172.21.15.79:6803/3615156451 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f35080692e0 con 0x7f34f403bef0 2024-04-23T16:41:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f34fd7fa640 1 -- 172.21.15.79:0/3557290425 <== osd.1 v2:172.21.15.79:6803/3615156451 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f35080692e0 con 0x7f34f403bef0 2024-04-23T16:41:44.545 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:44.545 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T16:41:44.545 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 >> v2:172.21.15.79:6803/3615156451 conn(0x7f34f403bef0 msgr2=0x7f34f403e310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.79:6803/3615156451 conn(0x7f34f403bef0 0x7f34f403e310 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 >> v2:172.21.15.204:6812/722020384 conn(0x7f34f40363e0 msgr2=0x7f34f40388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 --2- 172.21.15.79:0/3557290425 >> v2:172.21.15.204:6812/722020384 conn(0x7f34f40363e0 0x7f34f40388a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f350811e830 tx=0x7f34f8007e70 comp rx=0 tx=0).stop 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 10 monclient: shutdown 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 msgr2=0x7f35081cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 --2- 172.21.15.79:0/3557290425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f350809f910 0x7f35081cd290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f350401a060 tx=0x7f35040191d0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350d656640 1 -- 172.21.15.79:0/3557290425 reap_dead start 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 shutdown_connections 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 >> 172.21.15.79:0/3557290425 conn(0x7f35080a25b0 msgr2=0x7f35080710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 shutdown_connections 2024-04-23T16:41:44.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.542+0000 7f350f0b9640 1 -- 172.21.15.79:0/3557290425 wait complete. 2024-04-23T16:41:44.561 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.666+0000 7faa0a2d8640 10 monclient: get_monmap_and_config 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.666+0000 7faa0a2d8640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.666+0000 7faa0a2d8640 10 monclient: monmap: 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:44.672887+0000 2024-04-23T16:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:44.672887+0000 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 1 Processor -- start 2024-04-23T16:41:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 1 -- start start 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient: init 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa04134560 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa0409da60 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faa0413ab70 0x7faa0413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient(hunting): picked mon.noname-c con 0x7faa0413ab70 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa04135050 con 0x7faa04134560 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0409e8a0 con 0x7faa0409da60 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7faa0409d8a0 con 0x7faa0413ab70 2024-04-23T16:41:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa0a2d8640 10 monclient(hunting): authenticate will time out at 1058.860778s 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa03fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7faa04134560 0x7faa04134940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33458/0 (socket says 172.21.15.79:33458) 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa03fff640 1 -- 172.21.15.79:0/2011113391 learned_addr learned my addr 172.21.15.79:0/2011113391 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 1 --2- 172.21.15.79:0/2011113391 >> v2:172.21.15.204:3300/0 conn(0x7faa0413ab70 0x7faa0413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): get_auth_request con 0x7faa0413ab70 auth_method 0 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 1 -- 172.21.15.79:0/2011113391 >> v1:172.21.15.204:6789/0 conn(0x7faa0409da60 legacy=0x7faa04139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 1 -- 172.21.15.79:0/2011113391 >> v1:172.21.15.79:6789/0 conn(0x7faa04134560 legacy=0x7faa04134940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 1 monclient: found mon.noname-c 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:41:44.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.670+0000 7faa08875640 1 -- 172.21.15.79:0/2011113391 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa040a2ea0 con 0x7faa0413ab70 2024-04-23T16:41:44.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa08875640 1 -- 172.21.15.79:0/2011113391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 msgr2=0x7faa0413cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa08875640 1 --2- 172.21.15.79:0/2011113391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa0413cf60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009d90 tx=0x7fa9fc01e7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d0265945cad1c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 1 -- 172.21.15.79:0/2011113391 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc0aa700 con 0x7faa0413ab70 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: dump: 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 1 -- 172.21.15.79:0/2011113391 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9fc0aa8a0 con 0x7faa0413ab70 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 1 -- 172.21.15.79:0/2011113391 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc0aac70 con 0x7faa0413ab70 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa02ffd640 10 monclient: dump: 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.682 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.683 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa0a2d8640 5 monclient: authenticate success, global_id 4423 2024-04-23T16:41:44.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa0a2d8640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:44.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.674+0000 7faa0a2d8640 4 set_mon_vals no callback set 2024-04-23T16:41:44.688 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient: shutdown 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- 172.21.15.79:0/2011113391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 msgr2=0x7faa0413cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/2011113391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa0413cf60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009d90 tx=0x7fa9fc01e7e0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- 172.21.15.79:0/2011113391 shutdown_connections 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/2011113391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa0413cf60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- 172.21.15.79:0/2011113391 >> 172.21.15.79:0/2011113391 conn(0x7faa040a6250 msgr2=0x7faa040a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- 172.21.15.79:0/2011113391 shutdown_connections 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- 172.21.15.79:0/2011113391 wait complete. 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient: monmap: 2024-04-23T16:41:44.689 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 Processor -- start 2024-04-23T16:41:44.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- start start 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient: init 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa0409da60 0x7faa0412b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): picked mon.noname-a con 0x7faa0409da60 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 0x7faa0412b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): picked mon.noname-c con 0x7faa04134560 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa041329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): picked mon.noname-b con 0x7faa0413ab70 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faa0409c620 con 0x7faa0409da60 2024-04-23T16:41:44.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa0409c4a0 con 0x7faa0413ab70 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7faa0409c7a0 con 0x7faa04134560 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa0a2d8640 10 monclient(hunting): authenticate will time out at 1058.870270s 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 0x7faa0412b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 0x7faa0412b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45562/0 (socket says 172.21.15.79:45562) 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 -- 172.21.15.79:0/1048108565 learned_addr learned my addr 172.21.15.79:0/1048108565 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): get_auth_request con 0x7faa04134560 auth_method 0 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa0409da60 0x7faa0412b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): get_auth_request con 0x7faa0409da60 auth_method 0 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:44.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa08875640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa041329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa08875640 10 monclient(hunting): get_auth_request con 0x7faa0413ab70 auth_method 0 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa08875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa08875640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa08875640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 -- 172.21.15.79:0/1048108565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 msgr2=0x7faa041329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa041329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 -- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa0409da60 msgr2=0x7faa0412b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa0409da60 0x7faa0412b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:44.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 monclient: found mon.noname-c 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa037fe640 1 -- 172.21.15.79:0/1048108565 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa040722c0 con 0x7faa04134560 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.678+0000 7faa03fff640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa0409da60 0x7faa0412b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa037fe640 1 -- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 msgr2=0x7faa0412b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa037fe640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 0x7faa0412b8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400c450 tx=0x7fa9f400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4012a40 con 0x7faa04134560 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: dump: 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.695 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: _finish_auth 0 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:44.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 5 monclient: authenticate success, global_id 4442 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 monclient: _renew_subs 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa040702a0 con 0x7faa04134560 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 monclient: _renew_subs 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0406ffc0 con 0x7faa04134560 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f4017070 con 0x7faa04134560 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4012e40 con 0x7faa04134560 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:44.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 10 monclient: dump: 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:44.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa027fc640 4 set_mon_vals no callback set 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e4005090 con 0x7faa04134560 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fa9f401c460 con 0x7faa04134560 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 1 --2- 172.21.15.79:0/1048108565 >> v2:172.21.15.204:6812/722020384 conn(0x7fa9f0036410 0x7fa9f00388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fa9f4048c60 con 0x7faa04134560 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa03fff640 1 --2- 172.21.15.79:0/1048108565 >> v2:172.21.15.204:6812/722020384 conn(0x7fa9f0036410 0x7fa9f00388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:44.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.682+0000 7faa03fff640 10 monclient: get_auth_request con 0x7fa9f0036410 auth_method 0 2024-04-23T16:41:44.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.686+0000 7faa03fff640 1 --2- 172.21.15.79:0/1048108565 >> v2:172.21.15.204:6812/722020384 conn(0x7fa9f0036410 0x7fa9f00388d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8007ca0 tx=0x7fa9f8007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:44.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.686+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa9f401a520 con 0x7faa04134560 2024-04-23T16:41:44.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.686+0000 7faa00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:44.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.686+0000 7faa00ff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:44.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.846+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fa9e4000bd0 con 0x7fa9f0036410 2024-04-23T16:41:44.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.846+0000 7faa00ff9640 1 -- 172.21.15.79:0/1048108565 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa9e4000bd0 con 0x7fa9f0036410 2024-04-23T16:41:44.852 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 1 to deep-scrub 2024-04-23T16:41:44.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 >> v2:172.21.15.204:6812/722020384 conn(0x7fa9f0036410 msgr2=0x7fa9f00388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/1048108565 >> v2:172.21.15.204:6812/722020384 conn(0x7fa9f0036410 0x7fa9f00388d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8007ca0 tx=0x7fa9f8007bf0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 10 monclient: shutdown 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 msgr2=0x7faa0412b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 0x7faa0412b8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400c450 tx=0x7fa9f400c910 comp rx=0 tx=0).stop 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 shutdown_connections 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/1048108565 >> v2:172.21.15.204:6812/722020384 conn(0x7fa9f0036410 0x7fa9f00388d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa0413ab70 0x7faa041329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7faa04134560 0x7faa0412b8b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 --2- 172.21.15.79:0/1048108565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa0409da60 0x7faa0412b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 >> 172.21.15.79:0/1048108565 conn(0x7faa040a6250 msgr2=0x7faa04134940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 shutdown_connections 2024-04-23T16:41:44.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.850+0000 7faa0a2d8640 1 -- 172.21.15.79:0/1048108565 wait complete. 2024-04-23T16:41:44.871 INFO:tasks.ceph:Scrubbing osd.2 2024-04-23T16:41:44.871 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient: get_monmap_and_config 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient: build_initial_monmap 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient: monmap: 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:44.997912+0000 2024-04-23T16:41:44.997 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:44.997912+0000 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:44.998 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 Processor -- start 2024-04-23T16:41:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 -- start start 2024-04-23T16:41:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient: init 2024-04-23T16:41:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-23T16:41:45.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b040a3600 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 0x7f5b0413c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): picked mon.noname-b con 0x7f5b040a39e0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5b0413ce70 0x7f5b0413f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): picked mon.noname-c con 0x7f5b0413ce70 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b040a5780 con 0x7f5b040a3600 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5b040a4c80 con 0x7f5b0413ce70 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5b040a0f10 con 0x7f5b040a39e0 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b0ad03640 10 monclient(hunting): authenticate will time out at 1059.185755s 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 0x7f5b0413c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 0x7f5b0413c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45576/0 (socket says 172.21.15.79:45576) 2024-04-23T16:41:45.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 -- 172.21.15.79:0/35779253 learned_addr learned my addr 172.21.15.79:0/35779253 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): get_auth_request con 0x7f5b040a39e0 auth_method 0 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b092a0640 1 --2- 172.21.15.79:0/35779253 >> v2:172.21.15.204:3300/0 conn(0x7f5b0413ce70 0x7f5b0413f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b092a0640 10 monclient(hunting): get_auth_request con 0x7f5b0413ce70 auth_method 0 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b092a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b092a0640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b092a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient(hunting): handle_auth_done global_id 4445 payload 274 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 -- 172.21.15.79:0/35779253 >> v2:172.21.15.204:3300/0 conn(0x7f5b0413ce70 msgr2=0x7f5b0413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 --2- 172.21.15.79:0/35779253 >> v2:172.21.15.204:3300/0 conn(0x7f5b0413ce70 0x7f5b0413f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 -- 172.21.15.79:0/35779253 >> v1:172.21.15.204:6789/0 conn(0x7f5b040a3600 legacy=0x7f5b040a5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 monclient: found mon.noname-b 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.994+0000 7f5b03fff640 1 -- 172.21.15.79:0/35779253 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b040a18e0 con 0x7f5b040a39e0 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b03fff640 1 -- 172.21.15.79:0/35779253 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 msgr2=0x7f5b0413c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b03fff640 1 --2- 172.21.15.79:0/35779253 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 0x7f5b0413c930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5b040740f0 tx=0x7f5af401c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de7752e7ddee23bf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 1 -- 172.21.15.79:0/35779253 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b0070 con 0x7f5b040a39e0 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: dump: 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.007 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 1 -- 172.21.15.79:0/35779253 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af40ab600 con 0x7f5b040a39e0 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 5 monclient: authenticate success, global_id 4445 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:45.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 1 -- 172.21.15.79:0/35779253 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40ab9d0 con 0x7f5b040a39e0 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 4 set_mon_vals no callback set 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b037fe640 10 monclient: dump: 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.009 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 10 monclient: shutdown 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 -- 172.21.15.79:0/35779253 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 msgr2=0x7f5b0413c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 --2- 172.21.15.79:0/35779253 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 0x7f5b0413c930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5b040740f0 tx=0x7f5af401c370 comp rx=0 tx=0).stop 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 -- 172.21.15.79:0/35779253 shutdown_connections 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 --2- 172.21.15.79:0/35779253 >> v2:172.21.15.204:3300/0 conn(0x7f5b0413ce70 0x7f5b0413f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 --2- 172.21.15.79:0/35779253 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a39e0 0x7f5b0413c930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 -- 172.21.15.79:0/35779253 >> 172.21.15.79:0/35779253 conn(0x7f5b040aa370 msgr2=0x7f5b040a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 -- 172.21.15.79:0/35779253 shutdown_connections 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:44.998+0000 7f5b0ad03640 1 -- 172.21.15.79:0/35779253 wait complete. 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient: monmap: 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 Processor -- start 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 -- start start 2024-04-23T16:41:45.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient: init 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a3600 0x7f5b04139250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): picked mon.noname-c con 0x7f5b040a3600 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 0x7f5b04139790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): picked mon.noname-a con 0x7f5b040a39e0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b0413ce70 0x7f5b041cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): picked mon.noname-b con 0x7f5b0413ce70 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5b04141fc0 con 0x7f5b040a39e0 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b04142140 con 0x7f5b0413ce70 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5b041422c0 con 0x7f5b040a3600 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b0ad03640 10 monclient(hunting): authenticate will time out at 1059.193040s 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 0x7f5b04139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a3600 0x7f5b04139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 0x7f5b04139790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53760/0 (socket says 172.21.15.79:53760) 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 -- 172.21.15.79:0/684488393 learned_addr learned my addr 172.21.15.79:0/684488393 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): get_auth_request con 0x7f5b040a39e0 auth_method 0 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): get_auth_request con 0x7f5b040a3600 auth_method 0 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b092a0640 1 --2- 172.21.15.79:0/684488393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b0413ce70 0x7f5b041cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b092a0640 10 monclient(hunting): get_auth_request con 0x7f5b0413ce70 auth_method 0 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b092a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b092a0640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b092a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient(hunting): handle_auth_done global_id 14172 payload 824 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 -- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a3600 msgr2=0x7f5b04139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 --2- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a3600 0x7f5b04139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 -- 172.21.15.79:0/684488393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b0413ce70 msgr2=0x7f5b041cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 --2- 172.21.15.79:0/684488393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b0413ce70 0x7f5b041cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 monclient: found mon.noname-a 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 -- 172.21.15.79:0/684488393 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0413a690 con 0x7f5b040a39e0 2024-04-23T16:41:45.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b08a9f640 1 --2- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5b040a3600 0x7f5b04139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 -- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 msgr2=0x7f5b04139790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b03fff640 1 --2- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 0x7f5b04139790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5af401cc10 tx=0x7f5af401abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40ba020 con 0x7f5b040a39e0 2024-04-23T16:41:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.002+0000 7f5b017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: dump: 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 5 monclient: authenticate success, global_id 14172 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af40b6630 con 0x7f5b040a39e0 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 10 monclient: _renew_subs 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0413c1e0 con 0x7f5b040a39e0 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b0040 con 0x7f5b040a39e0 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: dump: 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.019 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b02ffd640 4 set_mon_vals no callback set 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 10 monclient: _renew_subs 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b040a4100 con 0x7f5b040a39e0 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f5af40bd020 con 0x7f5b040a39e0 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b040692e0 con 0x7f5b040a39e0 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.204:6812/722020384 conn(0x7f5ae40363e0 0x7f5ae40388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f5af40e31a0 con 0x7f5b040a39e0 2024-04-23T16:41:45.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.79:6800/3678369310 conn(0x7f5ae403bef0 0x7f5ae403e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 --> v2:172.21.15.79:6800/3678369310 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5af40ace50 con 0x7f5ae403bef0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5af40ea050 con 0x7f5b040a39e0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b017fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b092a0640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.79:6800/3678369310 conn(0x7f5ae403bef0 0x7f5ae403e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b092a0640 10 monclient: get_auth_request con 0x7f5ae403bef0 auth_method 0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b08a9f640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.204:6812/722020384 conn(0x7f5ae40363e0 0x7f5ae40388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.006+0000 7f5b08a9f640 10 monclient: get_auth_request con 0x7f5ae40363e0 auth_method 0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.010+0000 7f5b08a9f640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.204:6812/722020384 conn(0x7f5ae40363e0 0x7f5ae40388a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5aec007fd0 tx=0x7f5aec007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.010+0000 7f5b092a0640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.79:6800/3678369310 conn(0x7f5ae403bef0 0x7f5ae403e310 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.010+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== osd.2 v2:172.21.15.79:6800/3678369310 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5af40ace50 con 0x7f5ae403bef0 2024-04-23T16:41:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.022+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 --> v2:172.21.15.79:6800/3678369310 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5b0413bb10 con 0x7f5ae403bef0 2024-04-23T16:41:45.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b017fa640 1 -- 172.21.15.79:0/684488393 <== osd.2 v2:172.21.15.79:6800/3678369310 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5b0413bb10 con 0x7f5ae403bef0 2024-04-23T16:41:45.029 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:45.029 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T16:41:45.029 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 >> v2:172.21.15.79:6800/3678369310 conn(0x7f5ae403bef0 msgr2=0x7f5ae403e310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.79:6800/3678369310 conn(0x7f5ae403bef0 0x7f5ae403e310 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 >> v2:172.21.15.204:6812/722020384 conn(0x7f5ae40363e0 msgr2=0x7f5ae40388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 --2- 172.21.15.79:0/684488393 >> v2:172.21.15.204:6812/722020384 conn(0x7f5ae40363e0 0x7f5ae40388a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5aec007fd0 tx=0x7f5aec007e40 comp rx=0 tx=0).stop 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 10 monclient: shutdown 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 msgr2=0x7f5b04139790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 --2- 172.21.15.79:0/684488393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b040a39e0 0x7f5b04139790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5af401cc10 tx=0x7f5af401abb0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b092a0640 1 -- 172.21.15.79:0/684488393 reap_dead start 2024-04-23T16:41:45.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 shutdown_connections 2024-04-23T16:41:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 >> 172.21.15.79:0/684488393 conn(0x7f5b040aa370 msgr2=0x7f5b040a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 shutdown_connections 2024-04-23T16:41:45.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.026+0000 7f5b0ad03640 1 -- 172.21.15.79:0/684488393 wait complete. 2024-04-23T16:41:45.046 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-23T16:41:45.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient: get_monmap_and_config 2024-04-23T16:41:45.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient: monmap: 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:45.166346+0000 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:45.166346+0000 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:45.166 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:45.167 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:45.167 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:45.167 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 Processor -- start 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 -- start start 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient: init 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 0x7fcc641359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): picked mon.noname-a con 0x7fcc641355c0 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 0x7fcc640a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): picked mon.noname-b con 0x7fcc64135ee0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc6409c690 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc6409bdc0 con 0x7fcc6409c690 2024-04-23T16:41:45.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fcc64072bb0 con 0x7fcc641355c0 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcc6409f6c0 con 0x7fcc64135ee0 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc6a79e640 10 monclient(hunting): authenticate will time out at 1059.354242s 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 0x7fcc640a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 0x7fcc641359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 0x7fcc641359a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53772/0 (socket says 172.21.15.79:53772) 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): get_auth_request con 0x7fcc64135ee0 auth_method 0 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 1 -- 172.21.15.79:0/1645108302 learned_addr learned my addr 172.21.15.79:0/1645108302 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): get_auth_request con 0x7fcc641355c0 auth_method 0 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient(hunting): handle_auth_done global_id 4451 payload 274 2024-04-23T16:41:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 1 -- 172.21.15.79:0/1645108302 >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 msgr2=0x7fcc641359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 1 --2- 172.21.15.79:0/1645108302 >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 0x7fcc641359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 1 -- 172.21.15.79:0/1645108302 >> v1:172.21.15.204:6789/0 conn(0x7fcc6409c690 legacy=0x7fcc640a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 1 monclient: found mon.noname-b 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc637fe640 1 -- 172.21.15.79:0/1645108302 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6409f1e0 con 0x7fcc64135ee0 2024-04-23T16:41:45.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.162+0000 7fcc63fff640 1 --2- 172.21.15.79:0/1645108302 >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 0x7fcc641359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc637fe640 1 -- 172.21.15.79:0/1645108302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 msgr2=0x7fcc640a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc637fe640 1 --2- 172.21.15.79:0/1645108302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 0x7fcc640a13e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0097c0 tx=0x7fcc4c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae5a1fde8467cb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 1 -- 172.21.15.79:0/1645108302 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0b0070 con 0x7fcc64135ee0 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: dump: 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 1 -- 172.21.15.79:0/1645108302 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c0ab600 con 0x7fcc64135ee0 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 5 monclient: authenticate success, global_id 4451 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 1 -- 172.21.15.79:0/1645108302 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0a4e60 con 0x7fcc64135ee0 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 4 set_mon_vals no callback set 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc62ffd640 10 monclient: dump: 2024-04-23T16:41:45.177 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 10 monclient: shutdown 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 -- 172.21.15.79:0/1645108302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 msgr2=0x7fcc640a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/1645108302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 0x7fcc640a13e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0097c0 tx=0x7fcc4c01c300 comp rx=0 tx=0).stop 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 -- 172.21.15.79:0/1645108302 shutdown_connections 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/1645108302 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc64135ee0 0x7fcc640a13e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/1645108302 >> v2:172.21.15.79:3300/0 conn(0x7fcc641355c0 0x7fcc641359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 -- 172.21.15.79:0/1645108302 >> 172.21.15.79:0/1645108302 conn(0x7fcc640a6250 msgr2=0x7fcc640a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 -- 172.21.15.79:0/1645108302 shutdown_connections 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.166+0000 7fcc6a79e640 1 -- 172.21.15.79:0/1645108302 wait complete. 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient: monmap: 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:45.179 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 Processor -- start 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 -- start start 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient: init 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 0x7fcc6413c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): picked mon.noname-c con 0x7fcc6409c690 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc641355c0 0x7fcc64134700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): picked mon.noname-a con 0x7fcc641355c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc64135ee0 0x7fcc641cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): picked mon.noname-b con 0x7fcc64135ee0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:45.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6413ff00 con 0x7fcc641355c0 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6413fd80 con 0x7fcc64135ee0 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fcc64140080 con 0x7fcc6409c690 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc6a79e640 10 monclient(hunting): authenticate will time out at 1059.361295s 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 0x7fcc6413c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc641355c0 0x7fcc64134700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 0x7fcc6413c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45610/0 (socket says 172.21.15.79:45610) 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 -- 172.21.15.79:0/2291844768 learned_addr learned my addr 172.21.15.79:0/2291844768 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): get_auth_request con 0x7fcc6409c690 auth_method 0 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): get_auth_request con 0x7fcc641355c0 auth_method 0 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 -- 172.21.15.79:0/2291844768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc64135ee0 msgr2=0x7fcc641cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc64135ee0 0x7fcc641cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 -- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc641355c0 msgr2=0x7fcc64134700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc641355c0 0x7fcc64134700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 monclient: found mon.noname-c 2024-04-23T16:41:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc63fff640 1 -- 172.21.15.79:0/2291844768 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc641cc780 con 0x7fcc6409c690 2024-04-23T16:41:45.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.170+0000 7fcc637fe640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc641355c0 0x7fcc64134700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:45.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc63fff640 1 -- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 msgr2=0x7fcc6413c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc63fff640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 0x7fcc6413c3e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400efc0 tx=0x7fcc5400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54014680 con 0x7fcc6409c690 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: dump: 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.185 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc54019070 con 0x7fcc6409c690 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 5 monclient: authenticate success, global_id 4454 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 monclient: _renew_subs 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc641210b0 con 0x7fcc6409c690 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54014a20 con 0x7fcc6409c690 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 10 monclient: dump: 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.186 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc627fc640 4 set_mon_vals no callback set 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 monclient: _renew_subs 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6409dd20 con 0x7fcc6409c690 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fcc54012050 con 0x7fcc6409c690 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:45.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 1 --2- 172.21.15.79:0/2291844768 >> v2:172.21.15.204:6812/722020384 conn(0x7fcc440363e0 0x7fcc440388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc640692e0 con 0x7fcc6409c690 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fcc54049570 con 0x7fcc6409c690 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc637fe640 1 --2- 172.21.15.79:0/2291844768 >> v2:172.21.15.204:6812/722020384 conn(0x7fcc440363e0 0x7fcc440388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.174+0000 7fcc637fe640 10 monclient: get_auth_request con 0x7fcc440363e0 auth_method 0 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.178+0000 7fcc637fe640 1 --2- 172.21.15.79:0/2291844768 >> v2:172.21.15.204:6812/722020384 conn(0x7fcc440363e0 0x7fcc440388a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc6409c1d0 tx=0x7fcc4c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.178+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcc54010040 con 0x7fcc6409c690 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.178+0000 7fcc60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:45.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.178+0000 7fcc60ff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:45.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.338+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fcc6406ba20 con 0x7fcc440363e0 2024-04-23T16:41:45.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc60ff9640 1 -- 172.21.15.79:0/2291844768 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcc6406ba20 con 0x7fcc440363e0 2024-04-23T16:41:45.345 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 2 to deep-scrub 2024-04-23T16:41:45.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 >> v2:172.21.15.204:6812/722020384 conn(0x7fcc440363e0 msgr2=0x7fcc440388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/2291844768 >> v2:172.21.15.204:6812/722020384 conn(0x7fcc440363e0 0x7fcc440388a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc6409c1d0 tx=0x7fcc4c0a6000 comp rx=0 tx=0).stop 2024-04-23T16:41:45.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 10 monclient: shutdown 2024-04-23T16:41:45.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 msgr2=0x7fcc6413c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 0x7fcc6413c3e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400efc0 tx=0x7fcc5400c490 comp rx=0 tx=0).stop 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 shutdown_connections 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/2291844768 >> v2:172.21.15.204:6812/722020384 conn(0x7fcc440363e0 0x7fcc440388a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc64135ee0 0x7fcc641cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc641355c0 0x7fcc64134700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 --2- 172.21.15.79:0/2291844768 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fcc6409c690 0x7fcc6413c3e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 >> 172.21.15.79:0/2291844768 conn(0x7fcc640a6250 msgr2=0x7fcc640a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 shutdown_connections 2024-04-23T16:41:45.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.342+0000 7fcc6a79e640 1 -- 172.21.15.79:0/2291844768 wait complete. 2024-04-23T16:41:45.363 INFO:tasks.ceph:Scrubbing osd.3 2024-04-23T16:41:45.364 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient: get_monmap_and_config 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient: monmap: 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:45.489802+0000 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:45.489802+0000 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.489 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:45.490 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:45.490 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:45.490 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:45.490 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:45.490 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 1 Processor -- start 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 1 -- start start 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient: init 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 0x7f8c640a2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient(hunting): picked mon.noname-b con 0x7f8c640a3470 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c640a2e60 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:45.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c6413c9c0 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c640a3f60 con 0x7f8c640a2e60 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c640a1690 con 0x7f8c6413c9c0 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f8c640a0f30 con 0x7f8c640a3470 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6c297640 10 monclient(hunting): authenticate will time out at 1059.677640s 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c69832640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f8c640a2e60 0x7f8c640a3850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33474/0 (socket says 172.21.15.79:33474) 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 0x7f8c640a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c69832640 1 -- 172.21.15.79:0/3986895060 learned_addr learned my addr 172.21.15.79:0/3986895060 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): get_auth_request con 0x7f8c640a3470 auth_method 0 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 1 -- 172.21.15.79:0/3986895060 >> v1:172.21.15.204:6789/0 conn(0x7f8c6413c9c0 legacy=0x7f8c6413edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 1 -- 172.21.15.79:0/3986895060 >> v1:172.21.15.79:6789/0 conn(0x7f8c640a2e60 legacy=0x7f8c640a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 1 monclient: found mon.noname-b 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.486+0000 7f8c6a033640 1 -- 172.21.15.79:0/3986895060 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6406cc90 con 0x7f8c640a3470 2024-04-23T16:41:45.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6a033640 1 -- 172.21.15.79:0/3986895060 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 msgr2=0x7f8c640a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6a033640 1 --2- 172.21.15.79:0/3986895060 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 0x7f8c640a2920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0097f0 tx=0x7f8c5c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a288d030fe17f513 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 1 -- 172.21.15.79:0/3986895060 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0b0070 con 0x7f8c640a3470 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: dump: 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.496 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 1 -- 172.21.15.79:0/3986895060 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c5c0a97f0 con 0x7f8c640a3470 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 5 monclient: authenticate success, global_id 4457 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 1 -- 172.21.15.79:0/3986895060 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0a9bc0 con 0x7f8c640a3470 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 4 set_mon_vals no callback set 2024-04-23T16:41:45.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c69031640 10 monclient: dump: 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.498 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 10 monclient: shutdown 2024-04-23T16:41:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 -- 172.21.15.79:0/3986895060 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 msgr2=0x7f8c640a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 --2- 172.21.15.79:0/3986895060 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 0x7f8c640a2920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0097f0 tx=0x7f8c5c01c320 comp rx=0 tx=0).stop 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 -- 172.21.15.79:0/3986895060 shutdown_connections 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 --2- 172.21.15.79:0/3986895060 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a3470 0x7f8c640a2920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 -- 172.21.15.79:0/3986895060 >> 172.21.15.79:0/3986895060 conn(0x7f8c640aa4e0 msgr2=0x7f8c640a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 -- 172.21.15.79:0/3986895060 shutdown_connections 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.490+0000 7f8c6c297640 1 -- 172.21.15.79:0/3986895060 wait complete. 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient: monmap: 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 Processor -- start 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 -- start start 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient: init 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 0x7f8c641cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): picked mon.noname-c con 0x7f8c640a2e60 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c640a3470 0x7f8c6413b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): picked mon.noname-a con 0x7f8c640a3470 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c6413c9c0 0x7f8c64121410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): picked mon.noname-b con 0x7f8c6413c9c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64142f00 con 0x7f8c640a3470 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8c640a19b0 con 0x7f8c6413c9c0 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f8c64143080 con 0x7f8c640a2e60 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6c297640 10 monclient(hunting): authenticate will time out at 1059.685286s 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 0x7f8c641cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c640a3470 0x7f8c6413b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 0x7f8c641cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45620/0 (socket says 172.21.15.79:45620) 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 -- 172.21.15.79:0/600009464 learned_addr learned my addr 172.21.15.79:0/600009464 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): get_auth_request con 0x7f8c640a2e60 auth_method 0 2024-04-23T16:41:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): get_auth_request con 0x7f8c640a3470 auth_method 0 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a834640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c6413c9c0 0x7f8c64121410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient(hunting): handle_auth_done global_id 4460 payload 824 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 -- 172.21.15.79:0/600009464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c6413c9c0 msgr2=0x7f8c64121410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c6413c9c0 0x7f8c64121410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 -- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c640a3470 msgr2=0x7f8c6413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c640a3470 0x7f8c6413b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 monclient: found mon.noname-c 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a033640 1 -- 172.21.15.79:0/600009464 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c640a4190 con 0x7f8c640a2e60 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a834640 10 monclient: get_auth_request con 0x7f8c6413c9c0 auth_method 0 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c6a834640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c6413c9c0 0x7f8c64121410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.494+0000 7f8c69832640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c640a3470 0x7f8c6413b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:45.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6a033640 1 -- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 msgr2=0x7f8c641cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6a033640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 0x7f8c641cf400 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c01a9d0 tx=0x7f8c5c01ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0b0050 con 0x7f8c640a2e60 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: dump: 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.509 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c5c0bc070 con 0x7f8c640a2e60 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 5 monclient: authenticate success, global_id 4460 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 10 monclient: _renew_subs 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c64125350 con 0x7f8c640a2e60 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0b7450 con 0x7f8c640a2e60 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: dump: 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 10 monclient: _renew_subs 2024-04-23T16:41:45.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c68830640 4 set_mon_vals no callback set 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c64123060 con 0x7f8c640a2e60 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8c64068ae0 con 0x7f8c640a2e60 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f8c5c0a96b0 con 0x7f8c640a2e60 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6812/722020384 conn(0x7f8c500363e0 0x7f8c500388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f8c5c0e3c80 con 0x7f8c640a2e60 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6800/3194378740 conn(0x7f8c5003bef0 0x7f8c5003e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 --> v2:172.21.15.204:6800/3194378740 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c5c0c2e50 con 0x7f8c5003bef0 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8c5c0e82a0 con 0x7f8c640a2e60 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c62ffd640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6a834640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6800/3194378740 conn(0x7f8c5003bef0 0x7f8c5003e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c69832640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6812/722020384 conn(0x7f8c500363e0 0x7f8c500388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c69832640 10 monclient: get_auth_request con 0x7f8c500363e0 auth_method 0 2024-04-23T16:41:45.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.498+0000 7f8c6a834640 10 monclient: get_auth_request con 0x7f8c5003bef0 auth_method 0 2024-04-23T16:41:45.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.502+0000 7f8c69832640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6812/722020384 conn(0x7f8c500363e0 0x7f8c500388a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c64122c60 tx=0x7f8c58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.502+0000 7f8c6a834640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6800/3194378740 conn(0x7f8c5003bef0 0x7f8c5003e310 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.506+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== osd.3 v2:172.21.15.204:6800/3194378740 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c5c0c2e50 con 0x7f8c5003bef0 2024-04-23T16:41:45.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 --> v2:172.21.15.204:6800/3194378740 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8c64125130 con 0x7f8c5003bef0 2024-04-23T16:41:45.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c62ffd640 1 -- 172.21.15.79:0/600009464 <== osd.3 v2:172.21.15.204:6800/3194378740 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8c64125130 con 0x7f8c5003bef0 2024-04-23T16:41:45.524 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:45.524 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T16:41:45.524 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6800/3194378740 conn(0x7f8c5003bef0 msgr2=0x7f8c5003e310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6800/3194378740 conn(0x7f8c5003bef0 0x7f8c5003e310 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6812/722020384 conn(0x7f8c500363e0 msgr2=0x7f8c500388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 --2- 172.21.15.79:0/600009464 >> v2:172.21.15.204:6812/722020384 conn(0x7f8c500363e0 0x7f8c500388a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c64122c60 tx=0x7f8c58009040 comp rx=0 tx=0).stop 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 10 monclient: shutdown 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 msgr2=0x7f8c641cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 --2- 172.21.15.79:0/600009464 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f8c640a2e60 0x7f8c641cf400 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c01a9d0 tx=0x7f8c5c01ab60 comp rx=0 tx=0).stop 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6a834640 1 -- 172.21.15.79:0/600009464 reap_dead start 2024-04-23T16:41:45.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 shutdown_connections 2024-04-23T16:41:45.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 >> 172.21.15.79:0/600009464 conn(0x7f8c640aa4e0 msgr2=0x7f8c640a47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 shutdown_connections 2024-04-23T16:41:45.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.518+0000 7f8c6c297640 1 -- 172.21.15.79:0/600009464 wait complete. 2024-04-23T16:41:45.540 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.646+0000 7fb3d58b0640 10 monclient: get_monmap_and_config 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.646+0000 7fb3d58b0640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.646+0000 7fb3d58b0640 10 monclient: monmap: 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:45.653173+0000 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:45.653173+0000 2024-04-23T16:41:45.652 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:45.653 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 Processor -- start 2024-04-23T16:41:45.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 -- start start 2024-04-23T16:41:45.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient: init 2024-04-23T16:41:45.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb3d006cab0 0x7fb3d00a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d006cab0 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3d00a3c30 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb3d00a37e0 0x7fb3d00a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d00a37e0 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d0138590 con 0x7fb3d00a3c30 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d00a14f0 con 0x7fb3d006cab0 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d00a5750 con 0x7fb3d00a37e0 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3d58b0640 10 monclient(hunting): authenticate will time out at 1059.840729s 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb3d006cab0 0x7fb3d00a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7fb3d006cab0 0x7fb3d00a32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53808/0 (socket says 172.21.15.79:53808) 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 1 -- 172.21.15.79:0/4092179163 learned_addr learned my addr 172.21.15.79:0/4092179163 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3c67fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fb3d00a3c30 0x7fb3d00a4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33486/0 (socket says 172.21.15.79:33486) 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): get_auth_request con 0x7fb3d006cab0 auth_method 0 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 1 --2- 172.21.15.79:0/4092179163 >> v2:172.21.15.204:3300/0 conn(0x7fb3d00a37e0 0x7fb3d00a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): get_auth_request con 0x7fb3d00a37e0 auth_method 0 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_done global_id 4435 payload 274 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4092179163 >> v2:172.21.15.79:3300/0 conn(0x7fb3d006cab0 msgr2=0x7fb3d00a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 1 --2- 172.21.15.79:0/4092179163 >> v2:172.21.15.79:3300/0 conn(0x7fb3d006cab0 0x7fb3d00a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4092179163 >> v1:172.21.15.79:6789/0 conn(0x7fb3d00a3c30 legacy=0x7fb3d00a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 1 monclient: found mon.noname-c 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:41:45.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.650+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4092179163 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d00a17d0 con 0x7fb3d00a37e0 2024-04-23T16:41:45.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4092179163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 msgr2=0x7fb3d00a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3cf7fe640 1 --2- 172.21.15.79:0/4092179163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d00a4d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00a6f80 tx=0x7fb3c801c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ff3dac4657122e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 1 -- 172.21.15.79:0/4092179163 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80b2070 con 0x7fb3d00a37e0 2024-04-23T16:41:45.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: dump: 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 1 -- 172.21.15.79:0/4092179163 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d00a54c0 con 0x7fb3d00a37e0 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 1 -- 172.21.15.79:0/4092179163 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80a5980 con 0x7fb3d00a37e0 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 5 monclient: authenticate success, global_id 4435 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 4 set_mon_vals no callback set 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3ce7fc640 10 monclient: dump: 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.661 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.662 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.662 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.662 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.662 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.662 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 10 monclient: shutdown 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4092179163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 msgr2=0x7fb3d00a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4092179163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d00a4d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00a6f80 tx=0x7fb3c801c790 comp rx=0 tx=0).stop 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4092179163 shutdown_connections 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4092179163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d00a4d70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4092179163 >> v2:172.21.15.79:3300/0 conn(0x7fb3d006cab0 0x7fb3d00a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4092179163 >> 172.21.15.79:0/4092179163 conn(0x7fb3d00aa3d0 msgr2=0x7fb3d00a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4092179163 shutdown_connections 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.654+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4092179163 wait complete. 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: monmap: 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 Processor -- start 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 -- start start 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: init 2024-04-23T16:41:45.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 0x7fb3d01cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d006cab0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d0135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d00a37e0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 0x7fb3d013bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d00a3c30 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d0142590 con 0x7fb3d00a3c30 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d0142410 con 0x7fb3d00a37e0 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d0142710 con 0x7fb3d006cab0 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient(hunting): authenticate will time out at 1059.848997s 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 0x7fb3d01cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 0x7fb3d013bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 0x7fb3d01cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45624/0 (socket says 172.21.15.79:45624) 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 1 -- 172.21.15.79:0/4114568165 learned_addr learned my addr 172.21.15.79:0/4114568165 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): get_auth_request con 0x7fb3d006cab0 auth_method 0 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): get_auth_request con 0x7fb3d00a3c30 auth_method 0 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient(hunting): handle_auth_done global_id 14187 payload 824 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3c67fc640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d0135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 msgr2=0x7fb3d01cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 0x7fb3d01cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4114568165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 msgr2=0x7fb3d0135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d0135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 monclient: found mon.noname-a 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4114568165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d0136f80 con 0x7fb3d00a3c30 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ceffd640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 0x7fb3d01cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 -- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 msgr2=0x7fb3d013bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3cf7fe640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 0x7fb3d013bad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80ac000 tx=0x7fb3c8019530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80b2050 con 0x7fb3d00a3c30 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 10 monclient: dump: 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.672 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 5 monclient: authenticate success, global_id 14187 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: _renew_subs 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.658+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d00a4080 con 0x7fb3d00a3c30 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 10 monclient: _renew_subs 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d01d07c0 con 0x7fb3d00a3c30 2024-04-23T16:41:45.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c80b3040 con 0x7fb3d00a3c30 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80ad890 con 0x7fb3d00a3c30 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 10 monclient: dump: 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.674 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3cdffb640 4 set_mon_vals no callback set 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a8005090 con 0x7fb3d00a3c30 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fb3c80c0030 con 0x7fb3d00a3c30 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 1 --2- 172.21.15.79:0/4114568165 >> v2:172.21.15.204:6812/722020384 conn(0x7fb3b40363e0 0x7fb3b40388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fb3c80e3390 con 0x7fb3d00a3c30 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ceffd640 1 --2- 172.21.15.79:0/4114568165 >> v2:172.21.15.204:6812/722020384 conn(0x7fb3b40363e0 0x7fb3b40388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.662+0000 7fb3ceffd640 10 monclient: get_auth_request con 0x7fb3b40363e0 auth_method 0 2024-04-23T16:41:45.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.666+0000 7fb3ceffd640 1 --2- 172.21.15.79:0/4114568165 >> v2:172.21.15.204:6812/722020384 conn(0x7fb3b40363e0 0x7fb3b40388a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3b80112c0 tx=0x7fb3b8009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.666+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb3c80c7060 con 0x7fb3d00a3c30 2024-04-23T16:41:45.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.666+0000 7fb3ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:45.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.666+0000 7fb3ccff9640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:45.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.830+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fb3a8000bd0 con 0x7fb3b40363e0 2024-04-23T16:41:45.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.830+0000 7fb3ccff9640 1 -- 172.21.15.79:0/4114568165 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb3a8000bd0 con 0x7fb3b40363e0 2024-04-23T16:41:45.836 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 3 to deep-scrub 2024-04-23T16:41:45.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 >> v2:172.21.15.204:6812/722020384 conn(0x7fb3b40363e0 msgr2=0x7fb3b40388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4114568165 >> v2:172.21.15.204:6812/722020384 conn(0x7fb3b40363e0 0x7fb3b40388a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3b80112c0 tx=0x7fb3b8009240 comp rx=0 tx=0).stop 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 10 monclient: shutdown 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 msgr2=0x7fb3d013bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 0x7fb3d013bad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80ac000 tx=0x7fb3c8019530 comp rx=0 tx=0).stop 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 shutdown_connections 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4114568165 >> v2:172.21.15.204:6812/722020384 conn(0x7fb3b40363e0 0x7fb3b40388a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3d00a3c30 0x7fb3d013bad0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d00a37e0 0x7fb3d0135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 --2- 172.21.15.79:0/4114568165 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fb3d006cab0 0x7fb3d01cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 >> 172.21.15.79:0/4114568165 conn(0x7fb3d00aa3d0 msgr2=0x7fb3d009da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 shutdown_connections 2024-04-23T16:41:45.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.834+0000 7fb3d58b0640 1 -- 172.21.15.79:0/4114568165 wait complete. 2024-04-23T16:41:45.854 INFO:tasks.ceph:Scrubbing osd.4 2024-04-23T16:41:45.855 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-23T16:41:45.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient: get_monmap_and_config 2024-04-23T16:41:45.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient: monmap: 2024-04-23T16:41:45.973 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:45.974434+0000 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:45.974434+0000 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:45.974 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 Processor -- start 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 -- start start 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient: init 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f31dc0a3600 0x7f31dc0a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc0a3600 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f31dc0a5070 0x7f31dc137610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a5070 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31dc0a54c0 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31dc0a18e0 con 0x7f31dc0a54c0 2024-04-23T16:41:45.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f31dc0a4c80 con 0x7f31dc0a3600 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f31dc0a0f10 con 0x7f31dc0a5070 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31e23cf640 10 monclient(hunting): authenticate will time out at 1060.162300s 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f31dc0a3600 0x7f31dc0a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f31dc0a3600 0x7f31dc0a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53824/0 (socket says 172.21.15.79:53824) 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 1 -- 172.21.15.79:0/1956047867 learned_addr learned my addr 172.21.15.79:0/1956047867 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): get_auth_request con 0x7f31dc0a3600 auth_method 0 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 1 --2- 172.21.15.79:0/1956047867 >> v2:172.21.15.204:3300/0 conn(0x7f31dc0a5070 0x7f31dc137610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): get_auth_request con 0x7f31dc0a5070 auth_method 0 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient(hunting): handle_auth_done global_id 4438 payload 274 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 1 -- 172.21.15.79:0/1956047867 >> v2:172.21.15.79:3300/0 conn(0x7f31dc0a3600 msgr2=0x7f31dc0a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 1 --2- 172.21.15.79:0/1956047867 >> v2:172.21.15.79:3300/0 conn(0x7f31dc0a3600 0x7f31dc0a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 1 -- 172.21.15.79:0/1956047867 >> v1:172.21.15.79:6789/0 conn(0x7f31dc0a54c0 legacy=0x7f31dc137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:45.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 1 monclient: found mon.noname-c 2024-04-23T16:41:45.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:41:45.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.970+0000 7f31db7fe640 1 -- 172.21.15.79:0/1956047867 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0a1bc0 con 0x7f31dc0a5070 2024-04-23T16:41:45.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31db7fe640 1 -- 172.21.15.79:0/1956047867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 msgr2=0x7f31dc137610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31db7fe640 1 --2- 172.21.15.79:0/1956047867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 0x7f31dc137610 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0a4e00 tx=0x7f31c401c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69b32b5c3cfc5142 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 1 -- 172.21.15.79:0/1956047867 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c40b0070 con 0x7f31dc0a5070 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: dump: 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.982 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 1 -- 172.21.15.79:0/1956047867 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c40ab600 con 0x7f31dc0a5070 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 1 -- 172.21.15.79:0/1956047867 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c40ab9d0 con 0x7f31dc0a5070 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31daffd640 10 monclient: dump: 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 5 monclient: authenticate success, global_id 4438 2024-04-23T16:41:45.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:45.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 4 set_mon_vals no callback set 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 10 monclient: shutdown 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 -- 172.21.15.79:0/1956047867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 msgr2=0x7f31dc137610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 --2- 172.21.15.79:0/1956047867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 0x7f31dc137610 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0a4e00 tx=0x7f31c401c2e0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 -- 172.21.15.79:0/1956047867 shutdown_connections 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 --2- 172.21.15.79:0/1956047867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 0x7f31dc137610 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 --2- 172.21.15.79:0/1956047867 >> v2:172.21.15.79:3300/0 conn(0x7f31dc0a3600 0x7f31dc0a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 -- 172.21.15.79:0/1956047867 >> 172.21.15.79:0/1956047867 conn(0x7f31dc0aa370 msgr2=0x7f31dc0a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:45.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 -- 172.21.15.79:0/1956047867 shutdown_connections 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 -- 172.21.15.79:0/1956047867 wait complete. 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 10 monclient: build_initial_monmap 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 10 monclient: monmap: 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 Processor -- start 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 1 -- start start 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.978+0000 7f31e23cf640 10 monclient: init 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31dc0a3600 0x7f31dc1cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc0a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 0x7f31dc13e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc0a5070 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f31dc0a54c0 0x7f31dc09d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a54c0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc1422f0 con 0x7f31dc0a3600 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc142170 con 0x7f31dc0a5070 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc142470 con 0x7f31dc0a54c0 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient(hunting): authenticate will time out at 1060.172158s 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31dc0a3600 0x7f31dc1cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31dc0a3600 0x7f31dc1cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53840/0 (socket says 172.21.15.79:53840) 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 1 -- 172.21.15.79:0/1087972203 learned_addr learned my addr 172.21.15.79:0/1087972203 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): get_auth_request con 0x7f31dc0a3600 auth_method 0 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f31dc0a54c0 0x7f31dc09d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): get_auth_request con 0x7f31dc0a54c0 auth_method 0 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:45.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31db7fe640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 0x7f31dc13e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 -- 172.21.15.79:0/1087972203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 msgr2=0x7f31dc13e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0a5070 0x7f31dc13e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 -- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31dc0a3600 msgr2=0x7f31dc1cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31dc0a3600 0x7f31dc1cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:45.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient: _finish_hunting 0 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 monclient: found mon.noname-c 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 -- 172.21.15.79:0/1087972203 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc09f0f0 con 0x7f31dc0a54c0 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31dbfff640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31dc0a3600 0x7f31dc1cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 -- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f31dc0a54c0 msgr2=0x7f31dc09d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e096c640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f31dc0a54c0 0x7f31dc09d540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c410 tx=0x7f31cc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc017070 con 0x7f31dc0a54c0 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: dump: 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.993 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: _finish_auth 0 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: _check_auth_tickets 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc005230 con 0x7f31dc0a54c0 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 5 monclient: authenticate success, global_id 4466 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _renew_subs 2024-04-23T16:41:45.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc138b00 con 0x7f31dc0a54c0 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc0117a0 con 0x7f31dc0a54c0 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31d8ff9640 10 monclient: dump: 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:45.996 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:45.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31da7fc640 4 set_mon_vals no callback set 2024-04-23T16:41:45.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _renew_subs 2024-04-23T16:41:45.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc139d40 con 0x7f31dc0a54c0 2024-04-23T16:41:45.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.982+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f31dc1cfbb0 con 0x7f31dc0a54c0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f31cc01a070 con 0x7f31dc0a54c0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6812/722020384 conn(0x7f31bc036410 0x7f31bc0388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f31cc0481a0 con 0x7f31dc0a54c0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6804/3419064638 conn(0x7f31bc03bf20 0x7f31bc03e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 --> v2:172.21.15.204:6804/3419064638 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31bc03e9b0 con 0x7f31bc03bf20 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f31cc04f050 con 0x7f31dc0a54c0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31d8ff9640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31dbfff640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6812/722020384 conn(0x7f31bc036410 0x7f31bc0388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31dbfff640 10 monclient: get_auth_request con 0x7f31bc036410 auth_method 0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31db7fe640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6804/3419064638 conn(0x7f31bc03bf20 0x7f31bc03e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31db7fe640 10 monclient: get_auth_request con 0x7f31bc03bf20 auth_method 0 2024-04-23T16:41:45.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.986+0000 7f31dbfff640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6812/722020384 conn(0x7f31bc036410 0x7f31bc0388d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f31dc09ecf0 tx=0x7f31d400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.990+0000 7f31db7fe640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6804/3419064638 conn(0x7f31bc03bf20 0x7f31bc03e340 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:45.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:45.990+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== osd.4 v2:172.21.15.204:6804/3419064638 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31bc03e9b0 con 0x7f31bc03bf20 2024-04-23T16:41:46.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 --> v2:172.21.15.204:6804/3419064638 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f31dc0692e0 con 0x7f31bc03bf20 2024-04-23T16:41:46.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31d8ff9640 1 -- 172.21.15.79:0/1087972203 <== osd.4 v2:172.21.15.204:6804/3419064638 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f31dc0692e0 con 0x7f31bc03bf20 2024-04-23T16:41:46.010 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:46.011 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T16:41:46.011 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:46.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6804/3419064638 conn(0x7f31bc03bf20 msgr2=0x7f31bc03e340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6804/3419064638 conn(0x7f31bc03bf20 0x7f31bc03e340 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6812/722020384 conn(0x7f31bc036410 msgr2=0x7f31bc0388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 --2- 172.21.15.79:0/1087972203 >> v2:172.21.15.204:6812/722020384 conn(0x7f31bc036410 0x7f31bc0388d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f31dc09ecf0 tx=0x7f31d400b040 comp rx=0 tx=0).stop 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 10 monclient: shutdown 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f31dc0a54c0 msgr2=0x7f31dc09d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 --2- 172.21.15.79:0/1087972203 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f31dc0a54c0 0x7f31dc09d540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c410 tx=0x7f31cc00c8d0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e096c640 1 -- 172.21.15.79:0/1087972203 reap_dead start 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 shutdown_connections 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 >> 172.21.15.79:0/1087972203 conn(0x7f31dc0aa370 msgr2=0x7f31dc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 shutdown_connections 2024-04-23T16:41:46.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.006+0000 7f31e23cf640 1 -- 172.21.15.79:0/1087972203 wait complete. 2024-04-23T16:41:46.028 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-23T16:41:46.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.142+0000 7f598ed5d640 10 monclient: get_monmap_and_config 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.142+0000 7f598ed5d640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.142+0000 7f598ed5d640 10 monclient: monmap: 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:46.149442+0000 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:46.149442+0000 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:46.149 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:46.150 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:46.150 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:46.150 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:46.150 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 Processor -- start 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 -- start start 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient: init 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f59881355c0 0x7f59881359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): picked mon.noname-c con 0x7f59881355c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59880a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5988135ee0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f598809c690 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f598809bdc0 con 0x7f598809c690 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5988072bb0 con 0x7f59881355c0 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f598809f6c0 con 0x7f5988135ee0 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598ed5d640 10 monclient(hunting): authenticate will time out at 1060.337303s 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59880a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59880a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45648/0 (socket says 172.21.15.79:45648) 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 -- 172.21.15.79:0/1394699563 learned_addr learned my addr 172.21.15.79:0/1394699563 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): get_auth_request con 0x7f5988135ee0 auth_method 0 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598caf9640 1 --2- 172.21.15.79:0/1394699563 >> v2:172.21.15.204:3300/0 conn(0x7f59881355c0 0x7f59881359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598caf9640 10 monclient(hunting): get_auth_request con 0x7f59881355c0 auth_method 0 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598caf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598caf9640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f598caf9640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 -- 172.21.15.79:0/1394699563 >> v2:172.21.15.204:3300/0 conn(0x7f59881355c0 msgr2=0x7f59881359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 --2- 172.21.15.79:0/1394699563 >> v2:172.21.15.204:3300/0 conn(0x7f59881355c0 0x7f59881359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 -- 172.21.15.79:0/1394699563 >> v1:172.21.15.204:6789/0 conn(0x7f598809c690 legacy=0x7f59880a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 monclient: found mon.noname-b 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.146+0000 7f5987fff640 1 -- 172.21.15.79:0/1394699563 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598809f1e0 con 0x7f5988135ee0 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f5987fff640 1 -- 172.21.15.79:0/1394699563 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 msgr2=0x7f59880a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f5987fff640 1 --2- 172.21.15.79:0/1394699563 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59880a13e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f597c009870 tx=0x7f597c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd72c8c19fec6fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 1 -- 172.21.15.79:0/1394699563 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c0b0070 con 0x7f5988135ee0 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: dump: 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.160 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 1 -- 172.21.15.79:0/1394699563 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f597c0ab600 con 0x7f5988135ee0 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 1 -- 172.21.15.79:0/1394699563 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c0ab9d0 con 0x7f5988135ee0 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 5 monclient: authenticate success, global_id 4469 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 4 set_mon_vals no callback set 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f59877fe640 10 monclient: dump: 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 10 monclient: shutdown 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 -- 172.21.15.79:0/1394699563 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 msgr2=0x7f59880a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 --2- 172.21.15.79:0/1394699563 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59880a13e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f597c009870 tx=0x7f597c01c3c0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 -- 172.21.15.79:0/1394699563 shutdown_connections 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 --2- 172.21.15.79:0/1394699563 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59880a13e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 --2- 172.21.15.79:0/1394699563 >> v2:172.21.15.204:3300/0 conn(0x7f59881355c0 0x7f59881359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 -- 172.21.15.79:0/1394699563 >> 172.21.15.79:0/1394699563 conn(0x7f59880a6250 msgr2=0x7f59880a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 -- 172.21.15.79:0/1394699563 shutdown_connections 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 -- 172.21.15.79:0/1394699563 wait complete. 2024-04-23T16:41:46.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 10 monclient: monmap: 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.150+0000 7f598ed5d640 1 Processor -- start 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 -- start start 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient: init 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f598809c690 0x7f598811d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): picked mon.noname-b con 0x7f598809c690 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 0x7f5988134440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): picked mon.noname-a con 0x7f59881355c0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59881cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): picked mon.noname-c con 0x7f5988135ee0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f598813ff00 con 0x7f59881355c0 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f598813fd80 con 0x7f598809c690 2024-04-23T16:41:46.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5988140080 con 0x7f5988135ee0 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598ed5d640 10 monclient(hunting): authenticate will time out at 1060.344495s 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59881cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 0x7f5988134440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59881cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45658/0 (socket says 172.21.15.79:45658) 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 -- 172.21.15.79:0/2431499338 learned_addr learned my addr 172.21.15.79:0/2431499338 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): get_auth_request con 0x7f5988135ee0 auth_method 0 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 0x7f5988134440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53856/0 (socket says 172.21.15.79:53856) 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): get_auth_request con 0x7f59881355c0 auth_method 0 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598caf9640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f598809c690 0x7f598811d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598caf9640 10 monclient(hunting): get_auth_request con 0x7f598809c690 auth_method 0 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598caf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598caf9640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598caf9640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2024-04-23T16:41:46.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 -- 172.21.15.79:0/2431499338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f598809c690 msgr2=0x7f598811d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f598809c690 0x7f598811d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 -- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 msgr2=0x7f5988134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 0x7f5988134440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 monclient: found mon.noname-c 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 -- 172.21.15.79:0/2431499338 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598811eaf0 con 0x7f5988135ee0 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f5987fff640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 0x7f5988134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 -- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 msgr2=0x7f59881cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.154+0000 7f598d2fa640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59881cd540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f598000c7b0 tx=0x7f598000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980016020 con 0x7f5988135ee0 2024-04-23T16:41:46.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:46.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: dump: 2024-04-23T16:41:46.170 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.170 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f598000ac60 con 0x7f5988135ee0 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 5 monclient: authenticate success, global_id 4472 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 monclient: _renew_subs 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598811edd0 con 0x7f5988135ee0 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598000ce60 con 0x7f5988135ee0 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 10 monclient: dump: 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.172 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 monclient: _renew_subs 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5988072810 con 0x7f5988135ee0 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f5986ffd640 4 set_mon_vals no callback set 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f5980019020 con 0x7f5988135ee0 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f598811e730 con 0x7f5988135ee0 2024-04-23T16:41:46.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 1 --2- 172.21.15.79:0/2431499338 >> v2:172.21.15.204:6812/722020384 conn(0x7f5974036430 0x7f59740388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f5980049150 con 0x7f5988135ee0 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598caf9640 1 --2- 172.21.15.79:0/2431499338 >> v2:172.21.15.204:6812/722020384 conn(0x7f5974036430 0x7f59740388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.158+0000 7f598caf9640 10 monclient: get_auth_request con 0x7f5974036430 auth_method 0 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.162+0000 7f598caf9640 1 --2- 172.21.15.79:0/2431499338 >> v2:172.21.15.204:6812/722020384 conn(0x7f5974036430 0x7f59740388f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f597800b9d0 tx=0x7f5978008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.162+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f598811e730 con 0x7f5988135ee0 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.162+0000 7f59857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.162+0000 7f59857fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:46.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.322+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f59880692e0 con 0x7f5974036430 2024-04-23T16:41:46.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f59857fa640 1 -- 172.21.15.79:0/2431499338 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f59880692e0 con 0x7f5974036430 2024-04-23T16:41:46.329 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 4 to deep-scrub 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 >> v2:172.21.15.204:6812/722020384 conn(0x7f5974036430 msgr2=0x7f59740388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 --2- 172.21.15.79:0/2431499338 >> v2:172.21.15.204:6812/722020384 conn(0x7f5974036430 0x7f59740388f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f597800b9d0 tx=0x7f5978008040 comp rx=0 tx=0).stop 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 10 monclient: shutdown 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 msgr2=0x7f59881cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59881cd540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f598000c7b0 tx=0x7f598000cc70 comp rx=0 tx=0).stop 2024-04-23T16:41:46.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 shutdown_connections 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 --2- 172.21.15.79:0/2431499338 >> v2:172.21.15.204:6812/722020384 conn(0x7f5974036430 0x7f59740388f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5988135ee0 0x7f59881cd540 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59881355c0 0x7f5988134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 --2- 172.21.15.79:0/2431499338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f598809c690 0x7f598811d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 >> 172.21.15.79:0/2431499338 conn(0x7f59880a6250 msgr2=0x7f59880a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 shutdown_connections 2024-04-23T16:41:46.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.326+0000 7f598ed5d640 1 -- 172.21.15.79:0/2431499338 wait complete. 2024-04-23T16:41:46.346 INFO:tasks.ceph:Scrubbing osd.5 2024-04-23T16:41:46.346 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-23T16:41:46.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient: get_monmap_and_config 2024-04-23T16:41:46.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient: monmap: 2024-04-23T16:41:46.469 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:46.470246+0000 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:46.470246+0000 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:46.471 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 1 Processor -- start 2024-04-23T16:41:46.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 1 -- start start 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient: init 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec1349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient(hunting): picked mon.noname-b con 0x7f18ec1345d0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18ec134ef0 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18ec09f170 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ec0a22e0 con 0x7f18ec134ef0 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ec09c350 con 0x7f18ec09f170 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f18ec09c950 con 0x7f18ec1345d0 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18f14e9640 10 monclient(hunting): authenticate will time out at 1060.658088s 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec1349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18ea7fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f18ec134ef0 0x7f18ec0a14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33494/0 (socket says 172.21.15.79:33494) 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18ea7fc640 1 -- 172.21.15.79:0/436034032 learned_addr learned my addr 172.21.15.79:0/436034032 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): get_auth_request con 0x7f18ec1345d0 auth_method 0 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2024-04-23T16:41:46.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 1 -- 172.21.15.79:0/436034032 >> v1:172.21.15.204:6789/0 conn(0x7f18ec09f170 legacy=0x7f18ec0a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 1 -- 172.21.15.79:0/436034032 >> v1:172.21.15.79:6789/0 conn(0x7f18ec134ef0 legacy=0x7f18ec0a14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 1 monclient: found mon.noname-b 2024-04-23T16:41:46.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.466+0000 7f18eaffd640 1 -- 172.21.15.79:0/436034032 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ec09f810 con 0x7f18ec1345d0 2024-04-23T16:41:46.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18eaffd640 1 -- 172.21.15.79:0/436034032 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 msgr2=0x7f18ec1349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18eaffd640 1 --2- 172.21.15.79:0/436034032 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec1349b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009df0 tx=0x7f18e401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a41a047cbfd3b701 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 1 -- 172.21.15.79:0/436034032 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40ac070 con 0x7f18ec1345d0 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: dump: 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.480 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 5 monclient: authenticate success, global_id 4475 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 1 -- 172.21.15.79:0/436034032 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18e40a6750 con 0x7f18ec1345d0 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 4 set_mon_vals no callback set 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 1 -- 172.21.15.79:0/436034032 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40a6b20 con 0x7f18ec1345d0 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18e9ffb640 10 monclient: dump: 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.482 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 10 monclient: shutdown 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 -- 172.21.15.79:0/436034032 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 msgr2=0x7f18ec1349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 --2- 172.21.15.79:0/436034032 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec1349b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009df0 tx=0x7f18e401c7d0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 -- 172.21.15.79:0/436034032 shutdown_connections 2024-04-23T16:41:46.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 --2- 172.21.15.79:0/436034032 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec1349b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 -- 172.21.15.79:0/436034032 >> 172.21.15.79:0/436034032 conn(0x7f18ec0a25b0 msgr2=0x7f18ec0a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 -- 172.21.15.79:0/436034032 shutdown_connections 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.470+0000 7f18f14e9640 1 -- 172.21.15.79:0/436034032 wait complete. 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient: monmap: 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.484 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 Processor -- start 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 -- start start 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient: init 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 0x7f18ec133a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): picked mon.noname-a con 0x7f18ec09f170 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec09cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): picked mon.noname-c con 0x7f18ec1345d0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18ec134ef0 0x7f18ec09d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): picked mon.noname-b con 0x7f18ec134ef0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec141060 con 0x7f18ec09f170 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec140ee0 con 0x7f18ec134ef0 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f18ec1411e0 con 0x7f18ec1345d0 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18f14e9640 10 monclient(hunting): authenticate will time out at 1060.665241s 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 0x7f18ec133a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec09cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 0x7f18ec133a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53868/0 (socket says 172.21.15.79:53868) 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 -- 172.21.15.79:0/3610577591 learned_addr learned my addr 172.21.15.79:0/3610577591 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): get_auth_request con 0x7f18ec09f170 auth_method 0 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): get_auth_request con 0x7f18ec1345d0 auth_method 0 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient(hunting): handle_auth_done global_id 14199 payload 824 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 -- 172.21.15.79:0/3610577591 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 msgr2=0x7f18ec09cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 --2- 172.21.15.79:0/3610577591 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f18ec1345d0 0x7f18ec09cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 -- 172.21.15.79:0/3610577591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18ec134ef0 msgr2=0x7f18ec09d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 --2- 172.21.15.79:0/3610577591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18ec134ef0 0x7f18ec09d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 monclient: found mon.noname-a 2024-04-23T16:41:46.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 -- 172.21.15.79:0/3610577591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ec13dfd0 con 0x7f18ec09f170 2024-04-23T16:41:46.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 -- 172.21.15.79:0/3610577591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 msgr2=0x7f18ec133a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.474+0000 7f18eaffd640 1 --2- 172.21.15.79:0/3610577591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 0x7f18ec133a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18e401ae80 tx=0x7f18e4019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40c0070 con 0x7f18ec09f170 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: dump: 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.492 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18e40b1870 con 0x7f18ec09f170 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40ac050 con 0x7f18ec09f170 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 5 monclient: authenticate success, global_id 14199 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 10 monclient: _renew_subs 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18ec09e790 con 0x7f18ec09f170 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18e97fa640 4 set_mon_vals no callback set 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: dump: 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.494 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 10 monclient: _renew_subs 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18ec13b890 con 0x7f18ec09f170 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18ec0692e0 con 0x7f18ec09f170 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f18e40c3070 con 0x7f18ec09f170 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6812/722020384 conn(0x7f18d8036430 0x7f18d80388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f18e40e3880 con 0x7f18ec09f170 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6808/3961528928 conn(0x7f18d803bfc0 0x7f18d803e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 --> v2:172.21.15.204:6808/3961528928 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18d803ea50 con 0x7f18d803bfc0 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f18e40ea050 con 0x7f18ec09f170 2024-04-23T16:41:46.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18d37fe640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18ea7fc640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6812/722020384 conn(0x7f18d8036430 0x7f18d80388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18ea7fc640 10 monclient: get_auth_request con 0x7f18d8036430 auth_method 0 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18eb7fe640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6808/3961528928 conn(0x7f18d803bfc0 0x7f18d803e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.478+0000 7f18eb7fe640 10 monclient: get_auth_request con 0x7f18d803bfc0 auth_method 0 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.482+0000 7f18ea7fc640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6812/722020384 conn(0x7f18d8036430 0x7f18d80388f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18e0007ca0 tx=0x7f18e0007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.482+0000 7f18eb7fe640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6808/3961528928 conn(0x7f18d803bfc0 0x7f18d803e3e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.486+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== osd.5 v2:172.21.15.204:6808/3961528928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f18d803ea50 con 0x7f18d803bfc0 2024-04-23T16:41:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 --> v2:172.21.15.204:6808/3961528928 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f18ec133fb0 con 0x7f18d803bfc0 2024-04-23T16:41:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18d37fe640 1 -- 172.21.15.79:0/3610577591 <== osd.5 v2:172.21.15.204:6808/3961528928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f18ec133fb0 con 0x7f18d803bfc0 2024-04-23T16:41:46.503 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-23T16:41:46.503 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T16:41:46.503 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-23T16:41:46.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6808/3961528928 conn(0x7f18d803bfc0 msgr2=0x7f18d803e3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6808/3961528928 conn(0x7f18d803bfc0 0x7f18d803e3e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6812/722020384 conn(0x7f18d8036430 msgr2=0x7f18d80388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 --2- 172.21.15.79:0/3610577591 >> v2:172.21.15.204:6812/722020384 conn(0x7f18d8036430 0x7f18d80388f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18e0007ca0 tx=0x7f18e0007bf0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 10 monclient: shutdown 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 msgr2=0x7f18ec133a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 --2- 172.21.15.79:0/3610577591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18ec09f170 0x7f18ec133a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18e401ae80 tx=0x7f18e4019570 comp rx=0 tx=0).stop 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18eb7fe640 1 -- 172.21.15.79:0/3610577591 reap_dead start 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 shutdown_connections 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 >> 172.21.15.79:0/3610577591 conn(0x7f18ec0a25b0 msgr2=0x7f18ec0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 shutdown_connections 2024-04-23T16:41:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.498+0000 7f18f14e9640 1 -- 172.21.15.79:0/3610577591 wait complete. 2024-04-23T16:41:46.519 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-23T16:41:46.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.634+0000 7f7b32a98640 10 monclient: get_monmap_and_config 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.634+0000 7f7b32a98640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.634+0000 7f7b32a98640 10 monclient: monmap: 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:46.641131+0000 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:46.641131+0000 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:46.642 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 Processor -- start 2024-04-23T16:41:46.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 -- start start 2024-04-23T16:41:46.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient: init 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7b2c0a3600 0x7f7b2c0a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c0a3600 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b2c137c90 addr v1:172.21.15.204:6789/0 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b2c0a5d40 con 0x7f7b2c137c90 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7b2c0a0f10 con 0x7f7b2c0a3600 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2c0a3f20 con 0x7f7b2c138090 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b32a98640 10 monclient(hunting): authenticate will time out at 1060.828957s 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c0a4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45696/0 (socket says 172.21.15.79:45696) 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 -- 172.21.15.79:0/1275591010 learned_addr learned my addr 172.21.15.79:0/1275591010 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): get_auth_request con 0x7f7b2c138090 auth_method 0 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b30834640 1 --2- 172.21.15.79:0/1275591010 >> v2:172.21.15.204:3300/0 conn(0x7f7b2c0a3600 0x7f7b2c0a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient(hunting): handle_auth_done global_id 4481 payload 274 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 -- 172.21.15.79:0/1275591010 >> v2:172.21.15.204:3300/0 conn(0x7f7b2c0a3600 msgr2=0x7f7b2c0a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 --2- 172.21.15.79:0/1275591010 >> v2:172.21.15.204:3300/0 conn(0x7f7b2c0a3600 0x7f7b2c0a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 -- 172.21.15.79:0/1275591010 >> v1:172.21.15.204:6789/0 conn(0x7f7b2c137c90 legacy=0x7f7b2c0a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 monclient: found mon.noname-b 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b31035640 1 -- 172.21.15.79:0/1275591010 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c0a1600 con 0x7f7b2c138090 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b30834640 10 monclient: get_auth_request con 0x7f7b2c0a3600 auth_method 0 2024-04-23T16:41:46.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.638+0000 7f7b30834640 1 --2- 172.21.15.79:0/1275591010 >> v2:172.21.15.204:3300/0 conn(0x7f7b2c0a3600 0x7f7b2c0a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:41:46.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b31035640 1 -- 172.21.15.79:0/1275591010 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 msgr2=0x7f7b2c0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b31035640 1 --2- 172.21.15.79:0/1275591010 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c0a4ef0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009e50 tx=0x7f7b2401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2772e9ff352d309 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 1 -- 172.21.15.79:0/1275591010 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240a5640 con 0x7f7b2c138090 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: dump: 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 5 monclient: authenticate success, global_id 4481 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 1 -- 172.21.15.79:0/1275591010 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b240a57e0 con 0x7f7b2c138090 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:46.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 4 set_mon_vals no callback set 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 1 -- 172.21.15.79:0/1275591010 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240a5bb0 con 0x7f7b2c138090 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b2b7fe640 10 monclient: dump: 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: shutdown 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 -- 172.21.15.79:0/1275591010 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 msgr2=0x7f7b2c0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 --2- 172.21.15.79:0/1275591010 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c0a4ef0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009e50 tx=0x7f7b2401c8b0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 -- 172.21.15.79:0/1275591010 shutdown_connections 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 --2- 172.21.15.79:0/1275591010 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c0a4ef0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 --2- 172.21.15.79:0/1275591010 >> v2:172.21.15.204:3300/0 conn(0x7f7b2c0a3600 0x7f7b2c0a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 -- 172.21.15.79:0/1275591010 >> 172.21.15.79:0/1275591010 conn(0x7f7b2c0aa3e0 msgr2=0x7f7b2c0a68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 -- 172.21.15.79:0/1275591010 shutdown_connections 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 -- 172.21.15.79:0/1275591010 wait complete. 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: monmap: 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 Processor -- start 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 -- start start 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: init 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0a3600 0x7f7b2c138800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.642+0000 7f7b32a98640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c0a3600 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b2c137c90 0x7f7b2c138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c137c90 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c139280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c138090 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1422f0 con 0x7f7b2c0a3600 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c142170 con 0x7f7b2c137c90 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2c142470 con 0x7f7b2c138090 2024-04-23T16:41:46.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient(hunting): authenticate will time out at 1060.836063s 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c139280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:45702/0 (socket says 172.21.15.79:45702) 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 -- 172.21.15.79:0/691728630 learned_addr learned my addr 172.21.15.79:0/691728630 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): get_auth_request con 0x7f7b2c138090 auth_method 0 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0a3600 0x7f7b2c138800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): get_auth_request con 0x7f7b2c0a3600 auth_method 0 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b2bfff640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b2c137c90 0x7f7b2c138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b30834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient(hunting): handle_auth_done global_id 4484 payload 824 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 -- 172.21.15.79:0/691728630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b2c137c90 msgr2=0x7f7b2c138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b2c137c90 0x7f7b2c138d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 -- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0a3600 msgr2=0x7f7b2c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0a3600 0x7f7b2c138800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 monclient: found mon.noname-c 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 -- 172.21.15.79:0/691728630 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c13c620 con 0x7f7b2c138090 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b2bfff640 10 monclient: get_auth_request con 0x7f7b2c137c90 auth_method 0 2024-04-23T16:41:46.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b2bfff640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b2c137c90 0x7f7b2c138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 -- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 msgr2=0x7f7b2c139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b31035640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c139280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7b240af040 tx=0x7f7b24019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240bc070 con 0x7f7b2c138090 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: dump: 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.660 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b240b8850 con 0x7f7b2c138090 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240b8c20 con 0x7f7b2c138090 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 5 monclient: authenticate success, global_id 4484 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient: _renew_subs 2024-04-23T16:41:46.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2c13a330 con 0x7f7b2c138090 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b2affd640 4 set_mon_vals no callback set 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 10 monclient: dump: 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.662 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient: _renew_subs 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2c13c060 con 0x7f7b2c138090 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f7b240b6410 con 0x7f7b2c138090 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.646+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b2c068ae0 con 0x7f7b2c138090 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.650+0000 7f7b297fa640 1 --2- 172.21.15.79:0/691728630 >> v2:172.21.15.204:6812/722020384 conn(0x7f7b10036430 0x7f7b100388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.650+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f7b240e3580 con 0x7f7b2c138090 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.650+0000 7f7b30834640 1 --2- 172.21.15.79:0/691728630 >> v2:172.21.15.204:6812/722020384 conn(0x7f7b10036430 0x7f7b100388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.650+0000 7f7b30834640 10 monclient: get_auth_request con 0x7f7b10036430 auth_method 0 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.650+0000 7f7b30834640 1 --2- 172.21.15.79:0/691728630 >> v2:172.21.15.204:6812/722020384 conn(0x7f7b10036430 0x7f7b100388f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c13a000 tx=0x7f7b20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.654+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7b240c5020 con 0x7f7b2c138090 2024-04-23T16:41:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.654+0000 7f7b297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.654+0000 7f7b297fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:46.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.814+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f7b2c13c230 con 0x7f7b10036430 2024-04-23T16:41:46.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.814+0000 7f7b297fa640 1 -- 172.21.15.79:0/691728630 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7b2c13c230 con 0x7f7b10036430 2024-04-23T16:41:46.820 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 5 to deep-scrub 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 >> v2:172.21.15.204:6812/722020384 conn(0x7f7b10036430 msgr2=0x7f7b100388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 --2- 172.21.15.79:0/691728630 >> v2:172.21.15.204:6812/722020384 conn(0x7f7b10036430 0x7f7b100388f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c13a000 tx=0x7f7b20008040 comp rx=0 tx=0).stop 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 10 monclient: shutdown 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 msgr2=0x7f7b2c139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c139280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7b240af040 tx=0x7f7b24019650 comp rx=0 tx=0).stop 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 shutdown_connections 2024-04-23T16:41:46.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 --2- 172.21.15.79:0/691728630 >> v2:172.21.15.204:6812/722020384 conn(0x7f7b10036430 0x7f7b100388f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f7b2c138090 0x7f7b2c139280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b2c137c90 0x7f7b2c138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 --2- 172.21.15.79:0/691728630 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0a3600 0x7f7b2c138800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 >> 172.21.15.79:0/691728630 conn(0x7f7b2c0aa3e0 msgr2=0x7f7b2c136990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 shutdown_connections 2024-04-23T16:41:46.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.818+0000 7f7b32a98640 1 -- 172.21.15.79:0/691728630 wait complete. 2024-04-23T16:41:46.838 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient: get_monmap_and_config 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient: monmap: 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:41:46.959808+0000 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:41:46.959808+0000 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.959 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.960 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:41:46.960 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:41:46.960 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:41:46.960 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:41:46.960 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:41:46.960 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 Processor -- start 2024-04-23T16:41:46.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 -- start start 2024-04-23T16:41:46.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient: init 2024-04-23T16:41:46.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f70bc135a20 0x7f70bc135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f70bc135a20 addr v2:172.21.15.204:3300/0 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70bc09f430 addr v1:172.21.15.79:6789/0 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f70bc135300 0x7f70bc09c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f70bc135300 addr v2:172.21.15.79:3300/0 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70bc072f80 con 0x7f70bc09f430 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 -- --> v2:172.21.15.79:3300/0 -- mon_getmap magic: 0 -- 0x7f70bc09fb10 con 0x7f70bc135300 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f70bc09fdf0 con 0x7f70bc135a20 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c2a9b640 10 monclient(hunting): authenticate will time out at 1061.147632s 2024-04-23T16:41:46.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f70bc135300 0x7f70bc09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 1 --2- >> v2:172.21.15.79:3300/0 conn(0x7f70bc135300 0x7f70bc09c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47556/0 (socket says 172.21.15.79:47556) 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70bbfff640 1 --1- 172.21.15.79:0/141184322 >> v1:172.21.15.79:6789/0 conn(0x7f70bc09f430 0x7f70bc09ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:45718/0 (socket says 172.21.15.79:45718) 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 1 -- 172.21.15.79:0/141184322 learned_addr learned my addr 172.21.15.79:0/141184322 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 10 monclient(hunting): get_auth_request con 0x7f70bc135300 auth_method 0 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.954+0000 7f70c1038640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 1 --2- 172.21.15.79:0/141184322 >> v2:172.21.15.204:3300/0 conn(0x7f70bc135a20 0x7f70bc135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): get_auth_request con 0x7f70bc135a20 auth_method 0 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient(hunting): handle_auth_done global_id 4447 payload 274 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 1 -- 172.21.15.79:0/141184322 >> v2:172.21.15.79:3300/0 conn(0x7f70bc135300 msgr2=0x7f70bc09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 1 --2- 172.21.15.79:0/141184322 >> v2:172.21.15.79:3300/0 conn(0x7f70bc135300 0x7f70bc09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 1 -- 172.21.15.79:0/141184322 >> v1:172.21.15.79:6789/0 conn(0x7f70bc09f430 legacy=0x7f70bc09ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 1 monclient: found mon.noname-c 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.204:3300/0 2024-04-23T16:41:46.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.958+0000 7f70c0837640 1 -- 172.21.15.79:0/141184322 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70bc06de30 con 0x7f70bc135a20 2024-04-23T16:41:46.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c0837640 1 -- 172.21.15.79:0/141184322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc135a20 msgr2=0x7f70bc135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c0837640 1 --2- 172.21.15.79:0/141184322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc135a20 0x7f70bc135e00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70bc07b500 tx=0x7f70ac01c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28b2aa9d30f4556d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 1 -- 172.21.15.79:0/141184322 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0b0070 con 0x7f70bc135a20 2024-04-23T16:41:46.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: dump: 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 5 monclient: authenticate success, global_id 4447 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 1 -- 172.21.15.79:0/141184322 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ac0ab600 con 0x7f70bc135a20 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 10 monclient: get_monmap_and_config success 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 4 set_mon_vals no callback set 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 1 -- 172.21.15.79:0/141184322 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0ab9d0 con 0x7f70bc135a20 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70bb7fe640 10 monclient: dump: 2024-04-23T16:41:46.969 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.970 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 10 monclient: shutdown 2024-04-23T16:41:46.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:46.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 -- 172.21.15.79:0/141184322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc135a20 msgr2=0x7f70bc135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/141184322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc135a20 0x7f70bc135e00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70bc07b500 tx=0x7f70ac01c300 comp rx=0 tx=0).stop 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 -- 172.21.15.79:0/141184322 shutdown_connections 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/141184322 >> v2:172.21.15.79:3300/0 conn(0x7f70bc135300 0x7f70bc09c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/141184322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc135a20 0x7f70bc135e00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 -- 172.21.15.79:0/141184322 >> 172.21.15.79:0/141184322 conn(0x7f70bc0a25b0 msgr2=0x7f70bc0a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 -- 172.21.15.79:0/141184322 shutdown_connections 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.962+0000 7f70c2a9b640 1 -- 172.21.15.79:0/141184322 wait complete. 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient: build_initial_monmap 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient: monmap: 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:41:46.975 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 Processor -- start 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 -- start start 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient: init 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient: _reopen_session rank -1 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc09f430 0x7f70bc1cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): picked mon.noname-b con 0x7f70bc09f430 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f70bc135300 0x7f70bc134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f70bc135300 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:41:46.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 0x7f70bc134790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f70bc135a20 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f70bc1400b0 con 0x7f70bc135a20 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70bc09c860 con 0x7f70bc09f430 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): start opening mon connection 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f70bc140230 con 0x7f70bc135300 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): _renew_subs 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c2a9b640 10 monclient(hunting): authenticate will time out at 1061.157433s 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 0x7f70bc134790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f70bc135300 0x7f70bc134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 0x7f70bc134790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47572/0 (socket says 172.21.15.79:47572) 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 -- 172.21.15.79:0/284145754 learned_addr learned my addr 172.21.15.79:0/284145754 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): get_auth_request con 0x7f70bc135a20 auth_method 0 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): get_auth_request con 0x7f70bc135300 auth_method 0 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c0837640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc09f430 0x7f70bc1cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c0837640 10 monclient(hunting): get_auth_request con 0x7f70bc09f430 auth_method 0 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c0837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c0837640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c0837640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient(hunting): handle_auth_done global_id 14208 payload 824 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 -- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f70bc135300 msgr2=0x7f70bc134250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f70bc135300 0x7f70bc134250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 -- 172.21.15.79:0/284145754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc09f430 msgr2=0x7f70bc1cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc09f430 0x7f70bc1cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:46.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient: _finish_hunting 0 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 monclient: found mon.noname-a 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.966+0000 7f70c1038640 1 -- 172.21.15.79:0/284145754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70bc09d110 con 0x7f70bc135a20 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c1038640 1 -- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 msgr2=0x7f70bc134790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c1038640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 0x7f70bc134790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f70b4007ba0 tx=0x7f70b400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4018070 con 0x7f70bc135a20 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: dump: 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.980 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: _finish_auth 0 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: _check_auth_tickets 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b40039a0 con 0x7f70bc135a20 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4003d70 con 0x7f70bc135a20 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 10 monclient: dump: 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 5 monclient: authenticate success, global_id 14208 2024-04-23T16:41:46.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 monclient: _renew_subs 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70bc120c00 con 0x7f70bc135a20 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70baffd640 4 set_mon_vals no callback set 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 monclient: _renew_subs 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70bc120920 con 0x7f70bc135a20 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f70b401b070 con 0x7f70bc135a20 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70bc0692e0 con 0x7f70bc135a20 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 1 --2- 172.21.15.79:0/284145754 >> v2:172.21.15.204:6812/722020384 conn(0x7f70a80363e0 0x7f70a80388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f70b4048a40 con 0x7f70bc135a20 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c0837640 1 --2- 172.21.15.79:0/284145754 >> v2:172.21.15.204:6812/722020384 conn(0x7f70a80363e0 0x7f70a80388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:41:46.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.970+0000 7f70c0837640 10 monclient: get_auth_request con 0x7f70a80363e0 auth_method 0 2024-04-23T16:41:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.974+0000 7f70c0837640 1 --2- 172.21.15.79:0/284145754 >> v2:172.21.15.204:6812/722020384 conn(0x7f70a80363e0 0x7f70a80388a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f70bc06dbd0 tx=0x7f70ac0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:41:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.974+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f70b4010070 con 0x7f70bc135a20 2024-04-23T16:41:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.974+0000 7f70b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:41:46.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:46.974+0000 7f70b97fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:41:47.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.122+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f70bc09d9d0 con 0x7f70a80363e0 2024-04-23T16:41:47.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70b97fa640 1 -- 172.21.15.79:0/284145754 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33808 (secure 0 0 0) 0x7f70bc09d9d0 con 0x7f70a80363e0 2024-04-23T16:41:47.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:41:47.129 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:41:47.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 >> v2:172.21.15.204:6812/722020384 conn(0x7f70a80363e0 msgr2=0x7f70a80388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:47.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/284145754 >> v2:172.21.15.204:6812/722020384 conn(0x7f70a80363e0 0x7f70a80388a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f70bc06dbd0 tx=0x7f70ac0a6000 comp rx=0 tx=0).stop 2024-04-23T16:41:47.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 10 monclient: shutdown 2024-04-23T16:41:47.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:41:47.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 msgr2=0x7f70bc134790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 0x7f70bc134790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f70b4007ba0 tx=0x7f70b400d4a0 comp rx=0 tx=0).stop 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 shutdown_connections 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/284145754 >> v2:172.21.15.204:6812/722020384 conn(0x7f70a80363e0 0x7f70a80388a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f70bc135a20 0x7f70bc134790 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f70bc135300 0x7f70bc134250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 --2- 172.21.15.79:0/284145754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70bc09f430 0x7f70bc1cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 >> 172.21.15.79:0/284145754 conn(0x7f70bc0a25b0 msgr2=0x7f70bc0a39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 shutdown_connections 2024-04-23T16:41:47.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:41:47.126+0000 7f70c2a9b640 1 -- 172.21.15.79:0/284145754 wait complete. 2024-04-23T16:41:47.145 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-23T16:41:45.332209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163328,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067712,"statfs":{"total":579820584960,"available":579653337088,"internally_reserved":0,"allocated":1236992,"data_stored":1440242,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.909224+0000","last_change":"2024-04-23T16:39:43.715659+0000","last_active":"2024-04-23T16:40:08.909224+0000","last_peered":"2024-04-23T16:40:08.909224+0000","last_clean":"2024-04-23T16:40:08.909224+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:40:08.909224+0000","last_undegraded":"2024-04-23T16:40:08.909224+0000","last_fullsized":"2024-04-23T16:40:08.909224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:51.579465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.031042+0000","last_change":"2024-04-23T16:39:43.714945+0000","last_active":"2024-04-23T16:40:09.031042+0000","last_peered":"2024-04-23T16:40:09.031042+0000","last_clean":"2024-04-23T16:40:09.031042+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:40:09.031042+0000","last_undegraded":"2024-04-23T16:40:09.031042+0000","last_fullsized":"2024-04-23T16:40:09.031042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:02.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.908626+0000","last_change":"2024-04-23T16:39:43.708650+0000","last_active":"2024-04-23T16:40:08.908626+0000","last_peered":"2024-04-23T16:40:08.908626+0000","last_clean":"2024-04-23T16:40:08.908626+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:40:08.908626+0000","last_undegraded":"2024-04-23T16:40:08.908626+0000","last_fullsized":"2024-04-23T16:40:08.908626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:51.104410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.006900+0000","last_change":"2024-04-23T16:39:43.712301+0000","last_active":"2024-04-23T16:40:09.006900+0000","last_peered":"2024-04-23T16:40:09.006900+0000","last_clean":"2024-04-23T16:40:09.006900+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:40:09.006900+0000","last_undegraded":"2024-04-23T16:40:09.006900+0000","last_fullsized":"2024-04-23T16:40:09.006900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:08.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.039969+0000","last_change":"2024-04-23T16:39:43.713393+0000","last_active":"2024-04-23T16:40:09.039969+0000","last_peered":"2024-04-23T16:40:09.039969+0000","last_clean":"2024-04-23T16:40:09.039969+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:40:09.039969+0000","last_undegraded":"2024-04-23T16:40:09.039969+0000","last_fullsized":"2024-04-23T16:40:09.039969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:08.981001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.905773+0000","last_change":"2024-04-23T16:39:43.710172+0000","last_active":"2024-04-23T16:40:08.905773+0000","last_peered":"2024-04-23T16:40:08.905773+0000","last_clean":"2024-04-23T16:40:08.905773+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:40:08.905773+0000","last_undegraded":"2024-04-23T16:40:08.905773+0000","last_fullsized":"2024-04-23T16:40:08.905773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:02.147422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:09.032667+0000","last_change":"2024-04-23T16:39:43.719619+0000","last_active":"2024-04-23T16:40:09.032667+0000","last_peered":"2024-04-23T16:40:09.032667+0000","last_clean":"2024-04-23T16:40:09.032667+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:40:09.032667+0000","last_undegraded":"2024-04-23T16:40:09.032667+0000","last_fullsized":"2024-04-23T16:40:09.032667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:14:02.842378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.906573+0000","last_change":"2024-04-23T16:39:43.718645+0000","last_active":"2024-04-23T16:40:08.906573+0000","last_peered":"2024-04-23T16:40:08.906573+0000","last_clean":"2024-04-23T16:40:08.906573+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:40:08.906573+0000","last_undegraded":"2024-04-23T16:40:08.906573+0000","last_fullsized":"2024-04-23T16:40:08.906573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:42.695362+0000","last_clean_scrub_stamp":"2024-04-23T16:39:42.695362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:06:03.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":176,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:40:08.907996+0000","last_change":"2024-04-23T16:39:43.195510+0000","last_active":"2024-04-23T16:40:08.907996+0000","last_peered":"2024-04-23T16:40:08.907996+0000","last_clean":"2024-04-23T16:40:08.907996+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:40:08.907996+0000","last_undegraded":"2024-04-23T16:40:08.907996+0000","last_fullsized":"2024-04-23T16:40:08.907996+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:39:41.690496+0000","last_clean_scrub_stamp":"2024-04-23T16:39:41.690496+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:23.887071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607578,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607580,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607578,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640283,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:41:47.149 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.149 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.149 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.149 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.149 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.149 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.150 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.150 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-23T16:39:42.695362+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.150 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-23T16:39:41.690496+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=39, tm_sec=41, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=16, tm_min=41, tm_sec=42, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T16:41:47.150 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-23T16:42:07.151 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T16:42:07.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.281+0000 7fbf2713e640 10 monclient: get_monmap_and_config 2024-04-23T16:42:07.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.281+0000 7fbf2713e640 10 monclient: build_initial_monmap 2024-04-23T16:42:07.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.281+0000 7fbf2713e640 10 monclient: monmap: 2024-04-23T16:42:07.284 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:42:07.284 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:42:07.284350+0000 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:42:07.284350+0000 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:42:07.285 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.281+0000 7fbf2713e640 1 Processor -- start 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.281+0000 7fbf2713e640 1 -- start start 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.281+0000 7fbf2713e640 10 monclient: init 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient: _reopen_session rank -1 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 0x7fbf200a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2006ca30 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf200a32a0 addr v1:172.21.15.204:6789/0 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbf200a36a0 0x7fbf20137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf200a36a0 addr v2:172.21.15.204:3300/0 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf200a17b0 con 0x7fbf200a32a0 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbf2006ce10 con 0x7fbf200a36a0 2024-04-23T16:42:07.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbf200a14d0 con 0x7fbf2006ca30 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): _renew_subs 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf2713e640 10 monclient(hunting): authenticate will time out at 1081.472200s 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 0x7fbf200a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 0x7fbf200a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:57530/0 (socket says 172.21.15.79:57530) 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 -- 172.21.15.79:0/2362003926 learned_addr learned my addr 172.21.15.79:0/2362003926 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): get_auth_request con 0x7fbf2006ca30 auth_method 0 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:42:07.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf256db640 1 --2- 172.21.15.79:0/2362003926 >> v2:172.21.15.204:3300/0 conn(0x7fbf200a36a0 0x7fbf20137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf256db640 10 monclient(hunting): get_auth_request con 0x7fbf200a36a0 auth_method 0 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf256db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf256db640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf256db640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient(hunting): handle_auth_done global_id 4490 payload 274 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 -- 172.21.15.79:0/2362003926 >> v2:172.21.15.204:3300/0 conn(0x7fbf200a36a0 msgr2=0x7fbf20137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 --2- 172.21.15.79:0/2362003926 >> v2:172.21.15.204:3300/0 conn(0x7fbf200a36a0 0x7fbf20137cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 -- 172.21.15.79:0/2362003926 >> v1:172.21.15.204:6789/0 conn(0x7fbf200a32a0 legacy=0x7fbf201375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient: _finish_hunting 0 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 monclient: found mon.noname-b 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:42:07.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.285+0000 7fbf24eda640 1 -- 172.21.15.79:0/2362003926 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf200a1a90 con 0x7fbf2006ca30 2024-04-23T16:42:07.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf24eda640 1 -- 172.21.15.79:0/2362003926 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 msgr2=0x7fbf200a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:42:07.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf24eda640 1 --2- 172.21.15.79:0/2362003926 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 0x7fbf200a4c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009870 tx=0x7fbf1401c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe4529206449a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:42:07.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 1 -- 172.21.15.79:0/2362003926 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf140b0070 con 0x7fbf2006ca30 2024-04-23T16:42:07.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: dump: 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.295 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: _finish_auth 0 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: _check_auth_tickets 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 1 -- 172.21.15.79:0/2362003926 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf140ab600 con 0x7fbf2006ca30 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 5 monclient: authenticate success, global_id 4490 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: handle_config config(0 keys) 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: get_monmap_and_config success 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 1 -- 172.21.15.79:0/2362003926 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf140ab9d0 con 0x7fbf2006ca30 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 4 set_mon_vals no callback set 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf1f7fe640 10 monclient: dump: 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: shutdown 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:42:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- 172.21.15.79:0/2362003926 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 msgr2=0x7fbf200a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 --2- 172.21.15.79:0/2362003926 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 0x7fbf200a4c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009870 tx=0x7fbf1401c200 comp rx=0 tx=0).stop 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- 172.21.15.79:0/2362003926 shutdown_connections 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 --2- 172.21.15.79:0/2362003926 >> v2:172.21.15.204:3300/0 conn(0x7fbf200a36a0 0x7fbf20137cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 --2- 172.21.15.79:0/2362003926 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf2006ca30 0x7fbf200a4c80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- 172.21.15.79:0/2362003926 >> 172.21.15.79:0/2362003926 conn(0x7fbf200aa370 msgr2=0x7fbf200a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- 172.21.15.79:0/2362003926 shutdown_connections 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- 172.21.15.79:0/2362003926 wait complete. 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: build_initial_monmap 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: monmap: 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:42:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 Processor -- start 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- start start 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: init 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: _reopen_session rank -1 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf2006ca30 0x7fbf2013e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2006ca30 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf200a32a0 0x7fbf201385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf200a32a0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 0x7fbf20138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf200a36a0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20142100 con 0x7fbf200a36a0 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20138400 con 0x7fbf2006ca30 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7fbf20142280 con 0x7fbf200a32a0 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): _renew_subs 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.289+0000 7fbf2713e640 10 monclient(hunting): authenticate will time out at 1081.479957s 2024-04-23T16:42:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 0x7fbf20138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 0x7fbf20138af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39124/0 (socket says 172.21.15.79:39124) 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf200a32a0 0x7fbf201385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 -- 172.21.15.79:0/519532492 learned_addr learned my addr 172.21.15.79:0/519532492 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): get_auth_request con 0x7fbf200a36a0 auth_method 0 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): get_auth_request con 0x7fbf200a32a0 auth_method 0 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:42:07.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf24eda640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf2006ca30 0x7fbf2013e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf24eda640 10 monclient(hunting): get_auth_request con 0x7fbf2006ca30 auth_method 0 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf24eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf24eda640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf24eda640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient(hunting): handle_auth_done global_id 14211 payload 824 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 -- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf200a32a0 msgr2=0x7fbf201385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf200a32a0 0x7fbf201385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 -- 172.21.15.79:0/519532492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf2006ca30 msgr2=0x7fbf2013e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf2006ca30 0x7fbf2013e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient: _finish_hunting 0 2024-04-23T16:42:07.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 monclient: found mon.noname-a 2024-04-23T16:42:07.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:42:07.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.79:3300/0 2024-04-23T16:42:07.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 -- 172.21.15.79:0/519532492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2013aa60 con 0x7fbf200a36a0 2024-04-23T16:42:07.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf24eda640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf2006ca30 0x7fbf2013e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 -- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 msgr2=0x7fbf20138af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf256db640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 0x7fbf20138af0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000d800 tx=0x7fbf1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1000b8d0 con 0x7fbf200a36a0 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: dump: 2024-04-23T16:42:07.305 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: _finish_auth 0 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: _check_auth_tickets 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:42:07.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 5 monclient: authenticate success, global_id 14211 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf10019070 con 0x7fbf200a36a0 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: handle_config config(0 keys) 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1000bcb0 con 0x7fbf200a36a0 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 monclient: _renew_subs 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2013ace0 con 0x7fbf200a36a0 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1effd640 4 set_mon_vals no callback set 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 10 monclient: dump: 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.307 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 monclient: _renew_subs 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf200a4060 con 0x7fbf200a36a0 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:42:07.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.79:3300/0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf200692e0 con 0x7fbf200a36a0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.293+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7fbf1001e460 con 0x7fbf200a36a0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.297+0000 7fbf1d7fa640 1 --2- 172.21.15.79:0/519532492 >> v2:172.21.15.204:6812/722020384 conn(0x7fbf0c0363e0 0x7fbf0c0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.297+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7fbf100493b0 con 0x7fbf200a36a0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.297+0000 7fbf24eda640 1 --2- 172.21.15.79:0/519532492 >> v2:172.21.15.204:6812/722020384 conn(0x7fbf0c0363e0 0x7fbf0c0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.297+0000 7fbf24eda640 10 monclient: get_auth_request con 0x7fbf0c0363e0 auth_method 0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.297+0000 7fbf24eda640 1 --2- 172.21.15.79:0/519532492 >> v2:172.21.15.204:6812/722020384 conn(0x7fbf0c0363e0 0x7fbf0c0388a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf200a7930 tx=0x7fbf140a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.301+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbf1001e800 con 0x7fbf200a36a0 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.301+0000 7fbf1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:42:07.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.301+0000 7fbf1d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:42:07.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.453+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 --> v2:172.21.15.204:6812/722020384 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf2013f680 con 0x7fbf0c0363e0 2024-04-23T16:42:07.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.457+0000 7fbf1d7fa640 1 -- 172.21.15.79:0/519532492 <== mgr.4100 v2:172.21.15.204:6812/722020384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33814 (secure 0 0 0) 0x7fbf2013f680 con 0x7fbf0c0363e0 2024-04-23T16:42:07.460 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-23T16:42:07.460 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-23T16:42:07.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 >> v2:172.21.15.204:6812/722020384 conn(0x7fbf0c0363e0 msgr2=0x7fbf0c0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 --2- 172.21.15.79:0/519532492 >> v2:172.21.15.204:6812/722020384 conn(0x7fbf0c0363e0 0x7fbf0c0388a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf200a7930 tx=0x7fbf140a6000 comp rx=0 tx=0).stop 2024-04-23T16:42:07.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 10 monclient: shutdown 2024-04-23T16:42:07.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:42:07.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 msgr2=0x7fbf20138af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 0x7fbf20138af0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000d800 tx=0x7fbf1000dcc0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 shutdown_connections 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 --2- 172.21.15.79:0/519532492 >> v2:172.21.15.204:6812/722020384 conn(0x7fbf0c0363e0 0x7fbf0c0388a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf200a36a0 0x7fbf20138af0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7fbf200a32a0 0x7fbf201385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 --2- 172.21.15.79:0/519532492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf2006ca30 0x7fbf2013e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 >> 172.21.15.79:0/519532492 conn(0x7fbf200aa370 msgr2=0x7fbf200a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 shutdown_connections 2024-04-23T16:42:07.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.461+0000 7fbf2713e640 1 -- 172.21.15.79:0/519532492 wait complete. 2024-04-23T16:42:07.477 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-04-23T16:42:07.338950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163304,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067736,"statfs":{"total":579820584960,"available":579653361664,"internally_reserved":0,"allocated":1236992,"data_stored":1440242,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:47.192563+0000","last_change":"2024-04-23T16:41:47.192563+0000","last_active":"2024-04-23T16:41:47.192563+0000","last_peered":"2024-04-23T16:41:47.192563+0000","last_clean":"2024-04-23T16:41:47.192563+0000","last_became_active":"2024-04-23T16:39:43.715250+0000","last_became_peered":"2024-04-23T16:39:43.715250+0000","last_unstale":"2024-04-23T16:41:47.192563+0000","last_undegraded":"2024-04-23T16:41:47.192563+0000","last_fullsized":"2024-04-23T16:41:47.192563+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:47.192476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:47.192476+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.192476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:01.176281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:46.869342+0000","last_change":"2024-04-23T16:41:46.869342+0000","last_active":"2024-04-23T16:41:46.869342+0000","last_peered":"2024-04-23T16:41:46.869342+0000","last_clean":"2024-04-23T16:41:46.869342+0000","last_became_active":"2024-04-23T16:39:43.714307+0000","last_became_peered":"2024-04-23T16:39:43.714307+0000","last_unstale":"2024-04-23T16:41:46.869342+0000","last_undegraded":"2024-04-23T16:41:46.869342+0000","last_fullsized":"2024-04-23T16:41:46.869342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:46.869106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:46.869106+0000","last_clean_scrub_stamp":"2024-04-23T16:41:46.869106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:14:37.391859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:46.163262+0000","last_change":"2024-04-23T16:41:46.163262+0000","last_active":"2024-04-23T16:41:46.163262+0000","last_peered":"2024-04-23T16:41:46.163262+0000","last_clean":"2024-04-23T16:41:46.163262+0000","last_became_active":"2024-04-23T16:39:43.708108+0000","last_became_peered":"2024-04-23T16:39:43.708108+0000","last_unstale":"2024-04-23T16:41:46.163262+0000","last_undegraded":"2024-04-23T16:41:46.163262+0000","last_fullsized":"2024-04-23T16:41:46.163262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:46.163137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:46.163137+0000","last_clean_scrub_stamp":"2024-04-23T16:41:46.163137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:03.494053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:47.212230+0000","last_change":"2024-04-23T16:41:47.212230+0000","last_active":"2024-04-23T16:41:47.212230+0000","last_peered":"2024-04-23T16:41:47.212230+0000","last_clean":"2024-04-23T16:41:47.212230+0000","last_became_active":"2024-04-23T16:39:43.711239+0000","last_became_peered":"2024-04-23T16:39:43.711239+0000","last_unstale":"2024-04-23T16:41:47.212230+0000","last_undegraded":"2024-04-23T16:41:47.212230+0000","last_fullsized":"2024-04-23T16:41:47.212230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:47.211944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:47.211944+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.211944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:39:12.006215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:45.383773+0000","last_change":"2024-04-23T16:41:45.383773+0000","last_active":"2024-04-23T16:41:45.383773+0000","last_peered":"2024-04-23T16:41:45.383773+0000","last_clean":"2024-04-23T16:41:45.383773+0000","last_became_active":"2024-04-23T16:39:43.712890+0000","last_became_peered":"2024-04-23T16:39:43.712890+0000","last_unstale":"2024-04-23T16:41:45.383773+0000","last_undegraded":"2024-04-23T16:41:45.383773+0000","last_fullsized":"2024-04-23T16:41:45.383773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:45.383615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:45.383615+0000","last_clean_scrub_stamp":"2024-04-23T16:41:45.383615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:26.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:46.756102+0000","last_change":"2024-04-23T16:41:46.756163+0000","last_active":"2024-04-23T16:41:46.756102+0000","last_peered":"2024-04-23T16:41:46.756102+0000","last_clean":"2024-04-23T16:41:46.756102+0000","last_became_active":"2024-04-23T16:39:43.709392+0000","last_became_peered":"2024-04-23T16:39:43.709392+0000","last_unstale":"2024-04-23T16:41:46.756102+0000","last_undegraded":"2024-04-23T16:41:46.756102+0000","last_fullsized":"2024-04-23T16:41:46.756102+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:46.755890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:46.755890+0000","last_clean_scrub_stamp":"2024-04-23T16:41:46.755890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:48.396152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:46.022446+0000","last_change":"2024-04-23T16:41:46.022446+0000","last_active":"2024-04-23T16:41:46.022446+0000","last_peered":"2024-04-23T16:41:46.022446+0000","last_clean":"2024-04-23T16:41:46.022446+0000","last_became_active":"2024-04-23T16:39:43.719252+0000","last_became_peered":"2024-04-23T16:39:43.719252+0000","last_unstale":"2024-04-23T16:41:46.022446+0000","last_undegraded":"2024-04-23T16:41:46.022446+0000","last_fullsized":"2024-04-23T16:41:46.022446+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:46.022342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:46.022342+0000","last_clean_scrub_stamp":"2024-04-23T16:41:46.022342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:19:00.906788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:48.233395+0000","last_change":"2024-04-23T16:41:48.233395+0000","last_active":"2024-04-23T16:41:48.233395+0000","last_peered":"2024-04-23T16:41:48.233395+0000","last_clean":"2024-04-23T16:41:48.233395+0000","last_became_active":"2024-04-23T16:39:43.718044+0000","last_became_peered":"2024-04-23T16:39:43.718044+0000","last_unstale":"2024-04-23T16:41:48.233395+0000","last_undegraded":"2024-04-23T16:41:48.233395+0000","last_fullsized":"2024-04-23T16:41:48.233395+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:48.233141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:48.233141+0000","last_clean_scrub_stamp":"2024-04-23T16:41:48.233141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:14:38.125925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'119","reported_seq":187,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T16:41:45.127205+0000","last_change":"2024-04-23T16:41:45.127240+0000","last_active":"2024-04-23T16:41:45.127205+0000","last_peered":"2024-04-23T16:41:45.127205+0000","last_clean":"2024-04-23T16:41:45.127205+0000","last_became_active":"2024-04-23T16:39:43.193460+0000","last_became_peered":"2024-04-23T16:39:43.193460+0000","last_unstale":"2024-04-23T16:41:45.127205+0000","last_undegraded":"2024-04-23T16:41:45.127205+0000","last_fullsized":"2024-04-23T16:41:45.127205+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'119","last_scrub_stamp":"2024-04-23T16:41:45.127140+0000","last_deep_scrub":"21'119","last_deep_scrub_stamp":"2024-04-23T16:41:45.127140+0000","last_clean_scrub_stamp":"2024-04-23T16:41:45.127140+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:17:21.364733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11218,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607582,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607582,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":592659,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":188416,"data_stored":63731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5609,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:42:07.479 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.593+0000 7f5bc9bd2640 10 monclient: get_monmap_and_config 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.593+0000 7f5bc9bd2640 10 monclient: build_initial_monmap 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.593+0000 7f5bc9bd2640 10 monclient: monmap: 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:42:07.596473+0000 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:42:07.596473+0000 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a-legacy 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c-legacy 2024-04-23T16:42:07.596 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-b 2024-04-23T16:42:07.597 INFO:teuthology.orchestra.run.smithi079.stderr:3: v2:172.21.15.79:3300/0 mon.noname-a 2024-04-23T16:42:07.597 INFO:teuthology.orchestra.run.smithi079.stderr:4: v2:172.21.15.204:3300/0 mon.noname-c 2024-04-23T16:42:07.597 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.593+0000 7f5bc9bd2640 1 Processor -- start 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.593+0000 7f5bc9bd2640 1 -- start start 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.593+0000 7f5bc9bd2640 10 monclient: init 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient: _reopen_session rank -1 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5bc41355c0 0x7f5bc41359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc41355c0 addr v2:172.21.15.204:3300/0 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 0x7f5bc40a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc4135ee0 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bc409c690 addr v1:172.21.15.204:6789/0 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc409bdc0 con 0x7f5bc409c690 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc4072bb0 con 0x7f5bc41355c0 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc409f6c0 con 0x7f5bc4135ee0 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): _renew_subs 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc9bd2640 10 monclient(hunting): authenticate will time out at 1081.784307s 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 0x7f5bc40a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 0x7f5bc40a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:57554/0 (socket says 172.21.15.79:57554) 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 -- 172.21.15.79:0/3771809289 learned_addr learned my addr 172.21.15.79:0/3771809289 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): get_auth_request con 0x7f5bc4135ee0 auth_method 0 2024-04-23T16:42:07.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc37fe640 1 --2- 172.21.15.79:0/3771809289 >> v2:172.21.15.204:3300/0 conn(0x7f5bc41355c0 0x7f5bc41359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient(hunting): handle_auth_done global_id 4496 payload 274 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 -- 172.21.15.79:0/3771809289 >> v2:172.21.15.204:3300/0 conn(0x7f5bc41355c0 msgr2=0x7f5bc41359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 --2- 172.21.15.79:0/3771809289 >> v2:172.21.15.204:3300/0 conn(0x7f5bc41355c0 0x7f5bc41359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 -- 172.21.15.79:0/3771809289 >> v1:172.21.15.204:6789/0 conn(0x7f5bc409c690 legacy=0x7f5bc40a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient: _finish_hunting 0 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 monclient: found mon.noname-b 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc2ffd640 1 -- 172.21.15.79:0/3771809289 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc409f1e0 con 0x7f5bc4135ee0 2024-04-23T16:42:07.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc37fe640 10 monclient: get_auth_request con 0x7f5bc41355c0 auth_method 0 2024-04-23T16:42:07.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.597+0000 7f5bc37fe640 1 --2- 172.21.15.79:0/3771809289 >> v2:172.21.15.204:3300/0 conn(0x7f5bc41355c0 0x7f5bc41359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T16:42:07.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc2ffd640 1 -- 172.21.15.79:0/3771809289 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 msgr2=0x7f5bc40a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:42:07.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc2ffd640 1 --2- 172.21.15.79:0/3771809289 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 0x7f5bc40a13e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4009870 tx=0x7f5bb401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ebb2106a3fe8233 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:42:07.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 1 -- 172.21.15.79:0/3771809289 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40b0070 con 0x7f5bc4135ee0 2024-04-23T16:42:07.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-23T16:42:07.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: dump: 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: _finish_auth 0 2024-04-23T16:42:07.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: _check_auth_tickets 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 1 -- 172.21.15.79:0/3771809289 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb40ab600 con 0x7f5bc4135ee0 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 5 monclient: authenticate success, global_id 4496 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: handle_config config(0 keys) 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: get_monmap_and_config success 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 4 set_mon_vals no callback set 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 1 -- 172.21.15.79:0/3771809289 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40ab9d0 con 0x7f5bc4135ee0 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc27fc640 10 monclient: dump: 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.607 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: shutdown 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/3771809289 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 msgr2=0x7f5bc40a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/3771809289 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 0x7f5bc40a13e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4009870 tx=0x7f5bb401c3c0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/3771809289 shutdown_connections 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/3771809289 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc4135ee0 0x7f5bc40a13e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/3771809289 >> v2:172.21.15.204:3300/0 conn(0x7f5bc41355c0 0x7f5bc41359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/3771809289 >> 172.21.15.79:0/3771809289 conn(0x7f5bc40a6250 msgr2=0x7f5bc40a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/3771809289 shutdown_connections 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/3771809289 wait complete. 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: build_initial_monmap 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: monmap: 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.noname-a 2024-04-23T16:42:07.609 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.noname-b 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.noname-c 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 Processor -- start 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- start start 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: init 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: _reopen_session rank -1 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 0x7f5bc41cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc409c690 addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bc41355c0 0x7f5bc41cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc41355c0 addr [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bc4135ee0 0x7f5bc41cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc4135ee0 addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc413ff00 con 0x7f5bc4135ee0 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc413fd80 con 0x7f5bc41355c0 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): start opening mon connection 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 1 -- --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc4140080 con 0x7f5bc409c690 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): _renew_subs 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc9bd2640 10 monclient(hunting): authenticate will time out at 1081.791253s 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 0x7f5bc41cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 --2- >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 0x7f5bc41cd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3301/0 says I am v2:172.21.15.79:57558/0 (socket says 172.21.15.79:57558) 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bc4135ee0 0x7f5bc41cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 -- 172.21.15.79:0/12456115 learned_addr learned my addr 172.21.15.79:0/12456115 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): get_auth_request con 0x7f5bc409c690 auth_method 0 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): get_auth_request con 0x7f5bc4135ee0 auth_method 0 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc2ffd640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bc41355c0 0x7f5bc41cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 -- 172.21.15.79:0/12456115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bc41355c0 msgr2=0x7f5bc41cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bc41355c0 0x7f5bc41cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 -- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bc4135ee0 msgr2=0x7f5bc41cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bc4135ee0 0x7f5bc41cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient: _finish_hunting 0 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 monclient: found mon.noname-c 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc37fe640 1 -- 172.21.15.79:0/12456115 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc411ec90 con 0x7f5bc409c690 2024-04-23T16:42:07.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.601+0000 7f5bc3fff640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bc4135ee0 0x7f5bc41cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc37fe640 1 -- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 msgr2=0x7f5bc41cd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc37fe640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 0x7f5bc41cd4d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800c7f0 tx=0x7f5bb800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8007be0 con 0x7f5bc409c690 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: dump: 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.615 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: _finish_auth 0 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: _check_auth_tickets 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb8007d80 con 0x7f5bc409c690 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 5 monclient: authenticate success, global_id 4499 2024-04-23T16:42:07.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 monclient: _renew_subs 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc413cd90 con 0x7f5bc409c690 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: handle_config config(0 keys) 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8011630 con 0x7f5bc409c690 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 10 monclient: dump: 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 0469449d-bc82-4171-9959-1bb87fc3dec3 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-04-23T16:38:59.932761+0000 2024-04-23T16:42:07.617 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.c 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 monclient: _renew_subs 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc411d1c0 con 0x7f5bc409c690 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc1ffb640 4 set_mon_vals no callback set 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 4 ==== mgrmap(e 5) ==== 44289+0+0 (secure 0 0 0) 0x7f5bb80117d0 con 0x7f5bc409c690 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc411e840 con 0x7f5bc409c690 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 1 --2- 172.21.15.79:0/12456115 >> v2:172.21.15.204:6812/722020384 conn(0x7f5bb00363e0 0x7f5bb00388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3506+0+0 (secure 0 0 0) 0x7f5bb8049200 con 0x7f5bc409c690 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc2ffd640 1 --2- 172.21.15.79:0/12456115 >> v2:172.21.15.204:6812/722020384 conn(0x7f5bb00363e0 0x7f5bb00388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.605+0000 7f5bc2ffd640 10 monclient: get_auth_request con 0x7f5bb00363e0 auth_method 0 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.609+0000 7f5bc2ffd640 1 --2- 172.21.15.79:0/12456115 >> v2:172.21.15.204:6812/722020384 conn(0x7f5bb00363e0 0x7f5bb00388a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40ab000 tx=0x7f5bb40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.609+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5bb8015020 con 0x7f5bc409c690 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.609+0000 7f5babfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-23T16:42:07.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.609+0000 7f5babfff640 10 monclient: _finish_command 1 = system:0 2024-04-23T16:42:07.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.769+0000 7f5bc9bd2640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-23T16:42:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.769+0000 7f5bc9bd2640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-23T16:42:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.769+0000 7f5bc9bd2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.79:3301/0 2024-04-23T16:42:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.769+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 --> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f5bc406ba20 con 0x7f5bc409c690 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5bb8018710 con 0x7f5bc409c690 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5babfff640 10 monclient: handle_config config(1 keys) 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc1ffb640 4 set_mon_vals no callback set 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc1ffb640 10 set_mon_vals mon_health_to_clog = false 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5babfff640 1 -- 172.21.15.79:0/12456115 <== mon.2 v2:172.21.15.79:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f5bb8011df0 con 0x7f5bc409c690 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5babfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-23T16:42:07.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5babfff640 10 monclient: _finish_command 2 = system:0 2024-04-23T16:42:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 >> v2:172.21.15.204:6812/722020384 conn(0x7f5bb00363e0 msgr2=0x7f5bb00388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/12456115 >> v2:172.21.15.204:6812/722020384 conn(0x7f5bb00363e0 0x7f5bb00388a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40ab000 tx=0x7f5bb40a6000 comp rx=0 tx=0).stop 2024-04-23T16:42:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc9bd2640 10 monclient: shutdown 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc9bd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 msgr2=0x7f5bc41cd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.777+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 0x7f5bc41cd4d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800c7f0 tx=0x7f5bb800ccb0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 shutdown_connections 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/12456115 >> v2:172.21.15.204:6812/722020384 conn(0x7f5bb00363e0 0x7f5bb00388a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bc4135ee0 0x7f5bc41cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bc41355c0 0x7f5bc41cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 --2- 172.21.15.79:0/12456115 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5bc409c690 0x7f5bc41cd4d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 >> 172.21.15.79:0/12456115 conn(0x7f5bc40a6250 msgr2=0x7f5bc40a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 shutdown_connections 2024-04-23T16:42:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-23T16:42:07.781+0000 7f5bc9bd2640 1 -- 172.21.15.79:0/12456115 wait complete. 2024-04-23T16:42:07.796 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-23T16:42:07.796 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-23T16:42:07.797 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-23T16:42:07.797 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:13.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-23T16:42:13.898 INFO:tasks.ceph.osd.0:Stopped 2024-04-23T16:42:13.899 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-23T16:42:13.899 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:13.973 INFO:tasks.ceph.osd.1:Stopped 2024-04-23T16:42:13.973 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-23T16:42:13.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:18.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-23T16:42:18.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-23T16:42:20.075 INFO:tasks.ceph.osd.2:Stopped 2024-04-23T16:42:20.075 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-23T16:42:20.075 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:20.135 INFO:tasks.ceph.osd.3:Stopped 2024-04-23T16:42:20.136 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-23T16:42:20.136 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:24.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-23T16:42:24.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-23T16:42:26.237 INFO:tasks.ceph.osd.4:Stopped 2024-04-23T16:42:26.238 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-23T16:42:26.238 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:26.297 INFO:tasks.ceph.osd.5:Stopped 2024-04-23T16:42:26.297 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-23T16:42:26.297 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-23T16:42:26.298 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:26.327 INFO:tasks.ceph.mgr.x:Stopped 2024-04-23T16:42:26.327 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-23T16:42:26.327 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-23T16:42:26.328 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:26.350 INFO:tasks.ceph.mon.a:Stopped 2024-04-23T16:42:26.350 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-23T16:42:26.351 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:26.418 INFO:tasks.ceph.mon.c:Stopped 2024-04-23T16:42:26.419 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-23T16:42:26.419 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T16:42:26.433 INFO:tasks.ceph.mon.b:Stopped 2024-04-23T16:42:26.434 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-23T16:42:26.434 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | head -n 1 2024-04-23T16:42:26.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:42:26.477 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-23T16:42:26.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:42:26.667 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-23T16:42:26.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:42:26.850 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-23T16:42:27.006 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:42:27.007 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-23T16:42:27.179 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:42:27.179 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-23T16:42:27.378 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:42:27.378 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-23T16:42:27.546 INFO:tasks.ceph:Archiving mon data... 2024-04-23T16:42:27.547 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/data/mon.a.tgz 2024-04-23T16:42:27.548 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-04-23T16:42:27.553 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.7BxqP8apVs 2024-04-23T16:42:27.554 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7BxqP8apVs 2024-04-23T16:42:27.629 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.7BxqP8apVs 2024-04-23T16:42:27.734 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.7BxqP8apVs is 100KB 2024-04-23T16:42:27.749 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.7BxqP8apVs 2024-04-23T16:42:27.755 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/data/mon.c.tgz 2024-04-23T16:42:27.755 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-04-23T16:42:27.801 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.JDOZXRE0qt 2024-04-23T16:42:27.801 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JDOZXRE0qt 2024-04-23T16:42:27.877 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.JDOZXRE0qt 2024-04-23T16:42:27.991 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.JDOZXRE0qt is 101KB 2024-04-23T16:42:28.006 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.JDOZXRE0qt 2024-04-23T16:42:28.011 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/data/mon.b.tgz 2024-04-23T16:42:28.012 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-04-23T16:42:28.017 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.qcEkE98VZG 2024-04-23T16:42:28.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qcEkE98VZG 2024-04-23T16:42:28.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.qcEkE98VZG 2024-04-23T16:42:28.195 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.qcEkE98VZG is 101KB 2024-04-23T16:42:28.210 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.qcEkE98VZG 2024-04-23T16:42:28.215 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-23T16:42:28.215 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T16:42:28.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T16:42:28.279 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T16:42:28.323 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T16:42:28.323 INFO:tasks.ceph:Archiving crash dumps... 2024-04-23T16:42:28.324 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/remote/smithi079/crash 2024-04-23T16:42:28.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T16:42:28.338 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/remote/smithi204/crash 2024-04-23T16:42:28.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T16:42:28.378 INFO:tasks.ceph:Compressing logs... 2024-04-23T16:42:28.379 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T16:42:28.386 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T16:42:28.408 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27345.log 2024-04-23T16:42:28.408 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23834.log 2024-04-23T16:42:28.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27399.log 2024-04-23T16:42:28.409 INFO:teuthology.orchestra.run.smithi079.stderr: 89.1%/var/log/ceph/ceph-client.admin.23834.log: -- replaced with /var/log/ceph/ceph-client.admin.27345.log.gz 2024-04-23T16:42:28.409 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27962.log 2024-04-23T16:42:28.410 INFO:teuthology.orchestra.run.smithi079.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.23834.log.gz 2024-04-23T16:42:28.410 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26529.log 2024-04-23T16:42:28.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27399.log: /var/log/ceph/ceph-client.admin.27962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27007.log 2024-04-23T16:42:28.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26529.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27962.log.gz 2024-04-23T16:42:28.411 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27988.log 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.26529.log.gz 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.27636.log 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27007.log: /var/log/ceph/ceph-client.admin.27988.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27399.log.gz 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27636.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27988.log.gz 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27007.log.gz 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28066.log 2024-04-23T16:42:28.412 INFO:teuthology.orchestra.run.smithi079.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27636.log.gz 2024-04-23T16:42:28.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27936.log 2024-04-23T16:42:28.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27371.log 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27936.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28066.log.gz 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27011.log 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27936.log.gz 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.27371.log: -- /var/log/ceph/ceph-client.admin.26669.log 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27780.log 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26669.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27371.log.gz 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27011.log.gz 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27451.log 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26669.log.gz 2024-04-23T16:42:28.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27728.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26705.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.27780.log.gz 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27728.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27451.log.gz 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26705.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.27728.log.gz 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27041.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26705.log.gz 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27702.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27425.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27319.log 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27041.log.gz 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27702.log.gz 2024-04-23T16:42:28.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27884.log 2024-04-23T16:42:28.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27319.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27425.log.gz 2024-04-23T16:42:28.420 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26866.log 2024-04-23T16:42:28.420 INFO:teuthology.orchestra.run.smithi079.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27319.log.gz 2024-04-23T16:42:28.420 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-23T16:42:28.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27884.log: /var/log/ceph/ceph-client.admin.26866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26860.log 2024-04-23T16:42:28.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mon.a.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26866.log.gz 2024-04-23T16:42:28.425 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27503.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27884.log.gz 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27040.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28324.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.26860.log.gz 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28014.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27503.log.gz 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27040.log.gz 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28274.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26504.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28014.log: /var/log/ceph/ceph-client.admin.28274.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28324.log.gz 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27015.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28014.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27529.log 2024-04-23T16:42:28.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26504.log: 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28274.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27555.log 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26504.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27529.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20249.log 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27015.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27832.log 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27529.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.tmp-client.admin.20249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28300.log 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20249.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.27555.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27806.log 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26712.log 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27832.log.gz 2024-04-23T16:42:28.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28196.log 2024-04-23T16:42:28.428 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28300.log.gz 2024-04-23T16:42:28.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27910.log 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27806.log.gz 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28196.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26712.log.gz 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24028.log 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28196.log.gz 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27293.log 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27910.log.gz 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.24028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28118.log 2024-04-23T16:42:28.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26869.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24028.log.gz 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27047.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27293.log.gz 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26869.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28118.log.gz 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26555.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26714.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26869.log.gz 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26555.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27047.log.gz 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26714.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26555.log.gz 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27189.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26715.log 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.26714.log.gz/var/log/ceph/ceph-client.admin.27189.log: 2024-04-23T16:42:28.432 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26715.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27189.log.gz 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr: 91.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26715.log.gz 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26868.log 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28040.log 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26871.log 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27241.log 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26868.log.gz 2024-04-23T16:42:28.436 INFO:teuthology.orchestra.run.smithi079.stderr: 87.9%/var/log/ceph/ceph-client.admin.26871.log: -- replaced with /var/log/ceph/ceph-client.admin.28040.log.gz 2024-04-23T16:42:28.437 INFO:teuthology.orchestra.run.smithi079.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26871.log.gz 2024-04-23T16:42:28.437 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23954.log 2024-04-23T16:42:28.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27241.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T16:42:28.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.23954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27754.log 2024-04-23T16:42:28.437 INFO:teuthology.orchestra.run.smithi079.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.27241.log.gz 2024-04-23T16:42:28.437 INFO:teuthology.orchestra.run.smithi079.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.23954.log.gz 2024-04-23T16:42:28.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28170.log 2024-04-23T16:42:28.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28248.log 2024-04-23T16:42:28.438 INFO:teuthology.orchestra.run.smithi079.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T16:42:28.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28170.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.27754.log.gz 2024-04-23T16:42:28.439 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28170.log.gz 2024-04-23T16:42:28.439 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26856.log 2024-04-23T16:42:28.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26631.log 2024-04-23T16:42:28.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27163.log 2024-04-23T16:42:28.440 INFO:teuthology.orchestra.run.smithi079.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28248.log.gz 2024-04-23T16:42:28.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26631.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26856.log.gz 2024-04-23T16:42:28.440 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26631.log.gz 2024-04-23T16:42:28.441 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26709.log 2024-04-23T16:42:28.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27581.log 2024-04-23T16:42:28.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27858.log 2024-04-23T16:42:28.442 INFO:teuthology.orchestra.run.smithi079.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27163.log.gz 2024-04-23T16:42:28.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27581.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26709.log.gz 2024-04-23T16:42:28.443 INFO:teuthology.orchestra.run.smithi079.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27581.log.gz 2024-04-23T16:42:28.443 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26711.log 2024-04-23T16:42:28.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28222.log 2024-04-23T16:42:28.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28144.log 2024-04-23T16:42:28.445 INFO:teuthology.orchestra.run.smithi079.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27858.log.gz 2024-04-23T16:42:28.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28222.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26711.log.gz 2024-04-23T16:42:28.445 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28092.log 2024-04-23T16:42:28.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28144.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28222.log.gz 2024-04-23T16:42:28.446 INFO:teuthology.orchestra.run.smithi079.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28144.log.gz 2024-04-23T16:42:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27215.log 2024-04-23T16:42:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27477.log 2024-04-23T16:42:28.446 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28092.log: gzip/var/log/ceph/ceph-client.admin.27215.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27607.log 2024-04-23T16:42:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27477.log: 88.7% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28092.log.gz 2024-04-23T16:42:28.448 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27215.log.gz 2024-04-23T16:42:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26587.log 2024-04-23T16:42:28.448 INFO:teuthology.orchestra.run.smithi079.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27477.log.gz 2024-04-23T16:42:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27267.log 2024-04-23T16:42:28.450 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.26587.log: /var/log/ceph/ceph-client.admin.27267.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27607.log.gz 2024-04-23T16:42:28.450 INFO:teuthology.orchestra.run.smithi079.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26587.log.gz 2024-04-23T16:42:28.450 INFO:teuthology.orchestra.run.smithi079.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27267.log.gz 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26671.log 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26708.log 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26707.log 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.26671.log: /var/log/ceph/ceph-client.admin.26708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23901.log 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.26707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23925.log 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.26708.log.gz 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26672.log 2024-04-23T16:42:28.451 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.23901.log: /var/log/ceph/ceph-client.admin.23925.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.23925.log.gz 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.23901.log.gz 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.26672.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23949.log 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.5.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26707.log.gz 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26672.log.gz 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.23949.log: /var/log/ceph/ceph-mgr.x.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26671.log.gz 2024-04-23T16:42:28.453 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23973.log 2024-04-23T16:42:28.455 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T16:42:28.455 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.23949.log.gz 2024-04-23T16:42:28.455 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.23973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24004.log 2024-04-23T16:42:28.455 INFO:teuthology.orchestra.run.smithi204.stderr: 86.0% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph.audit.log: 2024-04-23T16:42:28.456 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26709.log 2024-04-23T16:42:28.456 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23973.log.gz 2024-04-23T16:42:28.456 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.24004.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-23T16:42:28.456 INFO:teuthology.orchestra.run.smithi204.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T16:42:28.456 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.26709.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-23T16:42:28.456 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24004.log.gz 2024-04-23T16:42:28.457 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23877.log 2024-04-23T16:42:28.457 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.23877.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.23877.log.gz 2024-04-23T16:42:28.457 INFO:teuthology.orchestra.run.smithi204.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.26709.log.gz 2024-04-23T16:42:28.489 INFO:teuthology.orchestra.run.smithi204.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-23T16:42:28.491 INFO:teuthology.orchestra.run.smithi079.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-23T16:42:28.501 INFO:teuthology.orchestra.run.smithi079.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-23T16:42:28.522 INFO:teuthology.orchestra.run.smithi079.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-23T16:42:28.548 INFO:teuthology.orchestra.run.smithi079.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-23T16:42:28.563 INFO:teuthology.orchestra.run.smithi204.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-23T16:42:28.575 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-23T16:42:28.580 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-23T16:42:28.954 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-23T16:42:28.955 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T16:42:28.955 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m0.524s 2024-04-23T16:42:28.955 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m0.898s 2024-04-23T16:42:28.956 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.048s 2024-04-23T16:42:28.974 INFO:teuthology.orchestra.run.smithi079.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-23T16:42:28.975 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-23T16:42:28.975 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m0.587s 2024-04-23T16:42:28.975 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m0.980s 2024-04-23T16:42:28.975 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.083s 2024-04-23T16:42:28.976 INFO:tasks.ceph:Archiving logs... 2024-04-23T16:42:28.976 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/remote/smithi079/log 2024-04-23T16:42:28.977 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T16:42:29.214 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/remote/smithi204/log 2024-04-23T16:42:29.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T16:42:29.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-23T16:42:29.347 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T16:42:29.355 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T16:42:29.355 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T16:42:29.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T16:42:29.436 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T16:42:29.450 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T16:42:29.450 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:42:29.453 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:42:29.464 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2024-04-23T16:42:29.464 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2024-04-23T16:42:29.464 INFO:teuthology.orchestra.run.smithi079.stdout: hv01.front.sepi 44.4.53.2 2 u 59 64 37 0.183 -3.886 0.700 2024-04-23T16:42:29.465 INFO:teuthology.orchestra.run.smithi079.stdout:*hv02.front.sepi 63.231.80.2 3 u 60 64 37 0.086 -1.228 0.950 2024-04-23T16:42:29.465 INFO:teuthology.orchestra.run.smithi079.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 37 0.085 -1.472 0.758 2024-04-23T16:42:29.465 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:42:29.467 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2024-04-23T16:42:29.467 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2024-04-23T16:42:29.468 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi 44.4.53.2 2 u 56 64 37 0.110 -2.823 0.438 2024-04-23T16:42:29.468 INFO:teuthology.orchestra.run.smithi204.stdout:*hv02.front.sepi 63.231.80.2 3 u 56 64 37 0.093 -0.649 0.691 2024-04-23T16:42:29.468 INFO:teuthology.orchestra.run.smithi204.stdout:+hv03.front.sepi 74.6.168.72 3 u 51 64 37 0.083 -0.972 0.630 2024-04-23T16:42:29.468 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T16:42:29.468 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T16:42:29.487 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T16:42:29.487 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T16:42:29.528 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T16:42:29.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T16:42:29.611 INFO:teuthology.task.internal:Duration was 754.736965 seconds 2024-04-23T16:42:29.611 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T16:42:29.643 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T16:42:29.643 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T16:42:29.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T16:42:29.684 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T16:42:29.684 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-04-23T16:42:29.684 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T16:42:29.730 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T16:42:29.730 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T16:42:29.741 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T16:42:29.741 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T16:42:29.774 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T16:42:29.810 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T16:42:29.811 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T16:42:29.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T16:42:29.905 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T16:42:29.919 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T16:42:29.920 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T16:42:29.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T16:42:29.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T16:42:29.961 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T16:42:29.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T16:42:29.985 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-04-23T16:42:30.001 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-04-23T16:42:30.021 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T16:42:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:42:30.045 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T16:42:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T16:42:30.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T16:42:30.078 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T16:42:30.079 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/remote/smithi079 2024-04-23T16:42:30.079 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T16:42:30.108 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669776/remote/smithi204 2024-04-23T16:42:30.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T16:42:30.130 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T16:42:30.130 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T16:42:30.151 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T16:42:30.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T16:42:30.189 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T16:42:30.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T16:42:30.216 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T16:42:30.216 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T16:42:30.219 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T16:42:30.223 INFO:teuthology.orchestra.run.smithi079.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 16:42 /home/ubuntu/cephtest 2024-04-23T16:42:30.225 INFO:teuthology.orchestra.run.smithi204.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 16:42 /home/ubuntu/cephtest 2024-04-23T16:42:30.226 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T16:42:30.248 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T16:42:30.315 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} duration: 754.7369651794434 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-23T16:42:30.315 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T16:42:30.421 INFO:teuthology.run:pass